Accelerated alignment of high resolution images and depth maps for low-bitwidth floating point representations

Abstract

An apparatus includes an interface and a processor. The interface can be configured to receive pixel data. The processor can be configured to (i) process the pixel data arranged as video frames, (ii) decompose each of the video frames including high resolution two-dimensional (2D) images into a plurality of 2D sub-images using a decomposition offset, (iii) decompose each of the video frames including high resolution depth map images into a plurality of depth map sub-images using the decomposition offset, (iv) compute a respective alignment matrix for each of the plurality of 2D sub-images and a respective one of the plurality of depth map sub-images using the decomposition offset, (v) perform matrix operations on the plurality of 2D sub-images and the plurality of depth map sub-images using the respective alignment matrices to obtain a plurality of aligned 2D sub-images and depth map sub-images, (v) determine projection coordinates and depth values for the plurality of aligned 2D sub-images and depth map sub-images based on the high resolution 2D images, and (vi) generate an aligned high resolution depth map using the projection coordinates and depth values.

Description

TECHNICAL FIELD

[0001] The present application relates generally to computer vision, and more particularly, to a method and / or apparatus for accelerated alignment of high resolution images and depth maps for low bit-width floating point representation. BACKGROUND

[0002] With the rapid development of machine vision, optical technology and artificial intelligence, the application and research of various depth map images have become an important branch of machine vision. An important application of depth map image research is the alignment between two-dimensional (2D) color (e.g., RGB) images and depth map images generated using various techniques, including time-of-flight (TOF), binocular stereo matching, monocular speckle structured light, etc. However, existing methods for alignment of high resolution RGB images and high resolution depth images still have some problems, such as not fast enough, or not accurate enough after acceleration, etc.

[0003] Depth 2D devices are generally implemented through embedded systems. Since the number of fractional bits of the floating point operation unit of many embedded processors is limited, when implementing existing depth alignment algorithms for high resolution images on an embedded platform, the resulting depth images will appear at locations with large pixel coordinate values. Therefore, the generated depth images cannot be accurately aligned with 2D images. The existing solution to this problem is to switch to a processor with a larger floating point mantissa. Doing so either increases the cost of the device or increases the time consumption of the algorithm and reduces the efficiency of the algorithm.

[0004] It would be desirable to implement accelerated alignment of high resolution images and depth maps for low bit-width floating point representation. SUMMARY

[0005] The present invention includes aspects relating to an apparatus comprising an interface and a processor. The interface can be configured to receive pixel data. The processor can be configured to (i) process the pixel data arranged as video frames, (ii) decompose each of the video frames comprising high resolution two-dimensional (2D) images into a plurality of 2D sub-images using a decomposition offset, (iii) decompose each of the video frames comprising high resolution depth map images into a plurality of depth map sub-images using the decomposition offset, (iv) compute a respective alignment matrix for each of the plurality of 2D sub-images and a respective one of the plurality of depth map sub-images using the decomposition offset, (v) perform matrix operations on the plurality of 2D sub-images and the plurality of depth map sub-images using the respective alignment matrices to obtain a plurality of aligned 2D sub-images and depth map sub-images, (v) determine projection coordinates and depth values for the plurality of aligned 2D sub-images and depth map sub-images based on the high resolution 2D images, and (vi) generate an aligned high resolution depth map using the projection coordinates and depth values. The video frames typically have pixel coordinate values that are larger than the number of bits supported by a floating point operator of the processor. The size of the plurality of 2D and depth map sub-images is typically determined based on the floating point operator of the processor.

[0006] In some embodiments of the above apparatus aspects, the matrix decomposition offset is configured to ensure that each of the sub-images is represented as a low-bit floating point representation without loss of precision.

[0007] In some embodiments of the above apparatus aspects, the processor can be configured to compute the matrix decomposition offset based on a number of significant bits of the low-bit floating point representation.

[0008] In some embodiments of the above apparatus aspects, the high resolution two-dimensional (2D) images and the high resolution depth map images are typically split into a plurality of rows and a plurality of columns of sub-images based on the decomposition offset.

[0009] In some embodiments of the above apparatus aspect, the processor can be further configured to compute a respective camera matrix for each of the 2D sub-images and each of the depth map sub-images. In some embodiments in which the processor computes the respective camera matrix, the processor can be further configured to compute a respective intermediate matrix for each of the respective 2D and depth map sub-images using the respective camera matrix of each respective 2D sub-image, the respective camera matrix of each respective depth map sub-image, and a camera projection matrix of the apparatus. In some embodiments, the processor can be further configured to compute a respective disparity-depth matrix for each of the respective 2D and depth map sub-images using the intermediate matrix. In some embodiments, the processor can be further configured to generate an aligned high resolution depth map for the high resolution 2D image using the respective disparity-depth matrices of the respective 2D and depth map sub-images.

[0010] In some embodiments of the above apparatus aspect, the processor can generally include a plurality of hardware engines configured to perform the matrix operations.

[0011] In some embodiments of the above apparatus aspect, the processor can include a system on a chip.

[0012] The present disclosure also includes an aspect directed to a method for accelerated alignment of 2D and depth images using low-bit-width floating-point representations, comprising: (i) receiving pixel data, (ii) processing the pixel data arranged as video frames, (iii) decomposing each of the video frames comprising a high resolution two-dimensional (2D) image into a plurality of 2D sub-images using a decomposition offset, (iv) decomposing each of the video frames comprising a high resolution depth map image into a plurality of depth map sub-images using the decomposition offset, (v) computing a respective alignment matrix for each of the plurality of 2D sub-images and a respective one of the plurality of depth map sub-images using the decomposition offset, (vi) performing matrix operations on the plurality of 2D sub-images and the plurality of depth map sub-images using the respective alignment matrices to obtain a plurality of aligned 2D sub-images and depth map sub-images, (vii) determining projection coordinates and depth values of the plurality of aligned 2D sub-images and depth map sub-images based on the high resolution 2D image, and (viii) generating an aligned high resolution depth map using the projection coordinates and depth values. The video frames can generally have pixel coordinate values that are larger than a number of bits supported by a floating-point operator of the processor, and the size of the plurality of 2D and depth map sub-images can be determined based on the floating-point operator of the processor.

[0013] In some embodiments of the above method aspect, the matrix decomposition offset can be configured to ensure that each of the sub-images is represented as a low-bit floating point representation without loss of precision.

[0014] In some embodiments of the above method aspect, the method can further include calculating the matrix decomposition offset based on a number of mantissa bits of the low-bit floating point representation.

[0015] In some embodiments of the above method aspect, the high resolution two-dimensional (2D) image and the high resolution depth map image can be split into sub-images of rows and columns based on the decomposition offset.

[0016] In some embodiments of the above method aspect, the method can further include calculating a respective camera matrix for each of the 2D sub-images and each of the depth map sub-images.

[0017] In some embodiments of the above method aspect, the method can further include calculating a respective intermediate matrix for each of the respective 2D and depth map sub-images using the respective camera matrix for each of the respective 2D sub-images, the respective camera matrix for each of the respective depth map sub-images, and a camera projection matrix of the device. In some embodiments, the method further includes calculating a respective disparity-depth matrix for each of the respective 2D and depth map sub-images using the intermediate matrix. In some embodiments, the method can further include generating an aligned high resolution depth map for the high resolution 2D image using the respective disparity-depth matrices for the respective 2D and depth map sub-images.

[0018] In some embodiments of the above method aspect, the method can further include performing the matrix operations with a plurality of hardware engines of a processor circuit. In some embodiments, the processor circuit can be part of an edge device. BRIEF DESCRIPTION OF DRAWINGS

[0019] Embodiments of the application will be apparent in view of the detailed description and the attached drawings, which follow.

[0020] Figure 1 FIG. 1 is a diagram illustrating an example of an edge device that can utilize a processor configured to enable accelerated alignment of high resolution images and depth maps of low-bit width floating point representations, in accordance with example embodiments of the present application.

[0021] Figure 2 FIG. 2 is a diagram illustrating an example camera that implements example embodiments of the present application.

[0022] Figure 3 FIG. 3 is a diagram illustrating a block diagram of a camera system.

[0023] Figure 4 is a diagram illustrating processing circuitry of a camera system implementing accelerated alignment of high resolution image and depth map for low bit-width floating point representation.

[0024] Figure 5 is a diagram illustrating depth estimation using a camera system.

[0025] Figure 6 is a diagram illustrating decomposition of high resolution 2D and depth map images into sub-images, which can be used to compute alignment matrices using low bit-width floating point representation.

[0026] Figure 7 is a diagram illustrating data arrangement for computing alignment matrices using low bit-width floating point representation.

[0027] Figure 8 is a flowchart illustrating a method for accelerating alignment of high resolution 2D image and high resolution depth map image for low bit-width floating point representation.

[0028] Figure 9 is an image illustrating an example of alignment results obtained using double precision floating point operations.

[0029] Figure 10 is an image illustrating an example of alignment results obtained using single precision floating point operations.

[0030] Figure 11 is an image illustrating an example of alignment results obtained using half precision floating point operations without using low bit-width representation based on matrix decomposition offset and lookup table combination.

[0031] Figure 12 is an image illustrating an example of alignment results obtained using half precision floating point operations using low bit-width representation based on matrix decomposition offset and lookup table combination. DETAILED DESCRIPTION

[0032] Embodiments of the present invention include providing accelerated alignment of high resolution image and depth map for low bit-width floating point representation that can (i) utilize low bit-width floating point representation based on matrix decomposition offset, (ii) decompose high resolution 2D image and high resolution depth map image into uniform sub-images of multiple rows and multiple columns based on the low bit-width floating point representation used, (iii) perform alignment operations on the corresponding 2D sub-images and depth sub-images to generate aligned depth images, (iv) be implemented in edge devices, (v) use depth images based on time-of-flight, binocular stereo matching, monocular speckle structured light, etc., (vi) maintain accuracy of alignment of high resolution 2D image and high resolution depth map image when using low bit-width floating point representation, (vii) provide faster alignment, and / or (viii) be implemented as one or more integrated circuits.

[0033] To solve the problem of depth alignment of high-resolution images in low-bit-width floating-point operation platforms, the present application designs and implements a depth alignment algorithm based on image splitting. The image splitting algorithm can eliminate the phenomenon that the depth image cannot be accurately aligned with the 2D image at the position where the pixel coordinate value is large due to the too small bit width of the floating-point operation platform in the prior art. Based on the combination of matrix decomposition, migration and table lookup, the alignment of high-resolution 2D images and high-resolution depth images is accelerated. In various embodiments, the original high-resolution image is decomposed into several uniform small images, and the alignment matrix of the several uniform small images is calculated through matrix decomposition offset to ensure that each decomposed image can be represented by low-bit-width representation without losing accuracy. Then, the several small images are respectively used for matrix operation with the alignment matrix obtained through matrix decomposition offset and low-bit-width representation to obtain the coordinate and depth value of the several small images, which are aligned based on the original high-resolution 2D image. The data is searched based on the coordinate and depth value to obtain the finally aligned high-resolution depth image.

[0034] Reference Figure 1 , Figure 1 A diagram showing an example of an edge device that can implement accelerated alignment of high-resolution 2D images and depth images for low-bit-width floating-point representation in accordance with example embodiments of the present application is shown. An overhead view of an area 50 is shown. In the example shown, the area 50 can be an outdoor location. Streets, vehicles, and buildings are shown.

[0035] Devices 100a-100n are shown at different locations in the area 50. Each of the devices 100a-100n can each implement an edge device. The edge devices 100a-100n can include smart IP cameras (e.g., camera systems). The edge devices 100a-100n can include low-power technology designed into embedded platforms deployed at the network edge (e.g., microprocessors running on sensors, cameras, or other battery-powered devices), where power consumption is a key concern. In examples, the edge devices 100a-100n can include various traffic cameras and intelligent transportation system (ITS) solutions.

[0036] The edge devices 100a-100n can be implemented for various applications. In the illustrated example, the edge devices 100a-100n can include an automatic number plate recognition (ANPR) camera 100a, a traffic camera 100b, a vehicle camera 100c, an access control camera 100d, an automated teller machine (ATM) camera 100e, a bullet camera 100f, a dome camera 100n, etc. In an example, the edge devices 100a-100n can be implemented as a traffic camera and intelligent transportation system (ITS) solution aimed at enhancing road safety with a combination of people and vehicle detection, vehicle make / model recognition, and automatic number plate recognition (ANPR) capabilities.

[0037] In the illustrated example, the area 50 can be an outdoor location. In some embodiments, the edge devices 100a-100n can be implemented at various indoor locations. In an example, the edge devices 100a-100n can incorporate convolutional neural networks for use in security (surveillance) applications and / or access control applications. In an example, the edge devices 100a-100n can implement accelerated alignment of high resolution 2D image and depth map images for low bit-width floating point representation for use in security (surveillance) applications and / or access control applications. In an example, the edge devices 100a-100n implemented as security cameras and access control applications can include battery-powered cameras, doorbell cameras, outdoor cameras, indoor cameras, etc. According to embodiments of the present application, security cameras and access control applications can realize performance advantages from applications that accelerate alignment of high resolution images and depth maps for low bit-width floating point representation. In an example, edge devices that utilize accelerated alignment of high resolution 2D and depth maps for low bit-width floating point representation according to embodiments of the present application can acquire large amounts of image data and perform on-device inferences to obtain useful information (e.g., multiple image time instances performed per network), thereby reducing bandwidth and / or reducing power consumption. The design, type, and / or application performed by the edge devices 100a-100n can vary according to design criteria of a particular implementation.

[0038] Reference is made to Figure 2 FIG. 1 illustrates a diagram of example edge device cameras that illustrate example embodiments implementing the present application. Camera systems 100a-100n are shown. Each camera device 100a-100n can have a different style and / or use case. For example, camera 100a can be a motion camera, camera 100b can be a ceiling mounted security camera, camera 100n can be a network camera, etc. Other types of cameras can be implemented (e.g., home security cameras, battery-powered cameras, doorbell cameras, stereo cameras, etc.). The design / style of the cameras 100a-100n can vary according to design criteria of a particular implementation.

[0039] Each of the camera systems 100a-100n can include a block (or circuit) 102, a block (or circuit) 104, and / or a block (or circuit) 106. The circuit 102 can implement a processor. The circuit 104 can implement a capture device. The circuit 106 can implement a structured light projector, a stereo camera, or a time-of-flight (ToF) sensor. The camera systems 100a-100n can include other components (not shown). Details of the components of the cameras 100a-100n can be described in connection with Figure 3

[0040] The processor 102 can be configured to perform matrix computations (e.g., dot products, matrix inversions, matrix transpositions, etc.). The processor 102 can be configured to implement an artificial neural network (ANN). In an example, the ANN can include a convolutional neural network (CNN). The processor 102 can be configured to implement a video encoder. The processor 102 can be configured to process pixel data arranged as video frames. The capture device 104 can be configured to capture pixel data that can be used by the processor 102 to generate video frames. When the circuit 106 implements a structured light projector, the structured light projector 106 can be configured to generate a structured light pattern (e.g., a speckle pattern). The structured light pattern can be projected onto a background (e.g., an environment). The capture device 104 can capture pixel data that includes an image of the background (e.g., the environment) with the speckle pattern. When the circuit 106 implements a time-of-flight sensor, the time-of-flight sensor 106 can be configured to generate a depth map (e.g., a point cloud).

[0041] The cameras 100a-100n can be edge devices. The processor 102 implemented by each of the cameras 100a-100n can enable the cameras 100a-100n to implement various functions internally (e.g., at a local level). For example, the processor 102 can be configured to perform object / event detection (e.g., computer vision operations), 3D reconstruction, video encoding, and / or video transcoding on the device. For example, the processor 102 can even perform advanced processes such as computer vision and 3D reconstruction without the need to upload video data to a cloud service to offload computationally intensive functions (e.g., computer vision, video encoding, video transcoding, etc.).

[0042] ​In some embodiments, multiple camera systems (e.g., camera systems 100a-100n) may be implemented, which can operate independently of each other. For example, each of cameras 100a-100n can individually analyze captured pixel data and perform event / object detection locally. In some embodiments, cameras 100a-100n may be configured as a camera network (e.g., security cameras that send video data to a central device such as network-attached storage and / or cloud services). The location and / or configuration of cameras 100a-100n may vary depending on the design criteria of the specific implementation.

[0043] The capture device 104 of each of the camera systems 100a-100n may include a single lens (e.g., a monocular camera). The processor 102 may be configured to accelerate the preprocessing of speckle structured light for monocular 3D reconstruction. Monocular 3D reconstruction can be performed to generate depth and / or parallax images without using a stereo camera. The processor 102 may also be configured to accelerate the alignment of high-resolution 2D images and high-resolution depth map images using low-bit-width floating-point representation.

[0044] refer to Figure 3 The diagram illustrates a block diagram of an example implementation of a camera system 100. In this example, the camera system 100 may include a processor / SoC 102, a capture device 104, and a structured light projector or time-of-flight sensor 106, as shown below. Figure 2The camera system 100 can also include a block (or circuit) 150, a block (or circuit) 152, a block (or circuit) 154, a block (or circuit) 156, a block (or circuit) 158, a block (or circuit) 160, a block (or circuit) 162, a block (or circuit) 164, and / or a block (or circuit) 166. The circuit 150 can implement a memory. The circuit 152 can implement a battery. The circuit 154 can implement a communication device. The circuit 156 can implement a wireless interface. The circuit 158 can implement a general purpose processor. The block 160 can implement an optical lens. The block 162 can implement a structured light pattern lens. The circuit 164 can implement one or more sensors. The circuit 166 can implement a human interface device (HID). In some embodiments, the camera system 100 can include the processor / SoC 102, the capture device 104, the IR structured light projector 106, the memory 150, the lens 160, the IR structured light projector 106, the structured light pattern lens 162, the sensor 164, the battery 152, the communication module 154, the wireless interface 156, and the processor 158. In another example, the camera system 100 can include the processor / SoC 102, the capture device 104, the structured light projector 106, the processor 158, the lens 160, the structured light pattern lens 162, and the sensor 164 as one device, and the memory 150, the battery 152, the communication module 154, and the wireless interface 156 can be components of separate devices. The camera system 100 can include other components (not shown). The number, type, and / or arrangement of components of the camera system 100 can vary depending on design criteria of a particular implementation.

[0045] The processor 102 can be implemented as a video processor. In an example, the processor 102 can be configured to receive a three-sensor video input with a high-speed SLVS / MIPI-CSI / LVCMOS interface. In some embodiments, the processor 102 can be configured to perform depth sensing in addition to generating video frames. In an example, the depth sensing can be performed in response to depth information and / or vector light data captured in the video frames.

[0046] The memory 150 can store data. The memory 150 can implement various types of memory, including but not limited to a cache, a flash memory, a memory card, a random access memory (RAM), a dynamic RAM (DRAM) memory, etc. The type and / or size of the memory 150 can vary depending on design criteria of a particular implementation. The data stored in the memory 150 can correspond to video files, motion information (e.g., readings from the sensor 164), video fusion parameters, image stabilization parameters, user inputs, computer vision models, feature sets, and / or metadata information. In some embodiments, the memory 150 can store reference images. The reference images can be used for computer vision operations, 3D reconstruction, etc.

[0047] The processor / SoC 102 can be configured to execute computer-readable code and / or process information. In various embodiments, the computer-readable code can be stored within the processor / SoC 102 (e.g., microcode, etc.) and / or the memory 150. In an example, the processor / SoC 102 can be configured to execute one or more artificial neural network models (e.g., a facial recognition CNN, an object detection CNN, an object classification CNN, a 3D reconstruction CNN, etc.) stored in the memory 150. In an example, the memory 150 can store one or more directed acyclic graphs (DAGs) and one or more sets of weights and biases that define one or more artificial neural network models. In an example, the processor / SoC 102 can be configured to execute instructions that implement a process for accelerated alignment of high resolution 2D image and depth map images with low bit-width floating point representations stored in the memory 150. The processor / SoC 102 can be configured to receive inputs from and / or present outputs to the memory 150. The processor / SoC 102 can be configured to present and / or receive other signals (not shown). The number and / or types of inputs and / or outputs of the processor / SoC 102 can vary depending on the design criteria for a particular implementation. The processor / SoC 102 can be configured for low power (e.g., battery) operation.

[0048] The battery 152 can be configured to store and / or supply power for components of the camera system 100. Dynamic driver mechanisms for rolling shutter sensors can be configured to conserve power consumption. Reducing power consumption can enable the camera system 100 to operate for extended periods of time using the battery 152 without requiring recharging. The battery 152 can be rechargeable. The battery 152 can be built-in (e.g., non- replaceable) or replaceable. The battery 152 can have an input for connecting to an external power source (e.g., for charging). In some embodiments, the apparatus 100 can be powered by an external power source (e.g., the battery 152 can not be implemented or can be implemented as a backup power source). Various battery technologies and / or chemistries can be used to implement the battery 152. The type of battery 152 implemented can vary depending on the design criteria for a particular implementation.

[0049] The communication module 154 can be configured to implement one or more communication protocols. For example, the communication module 154 and the wireless interface 156 can be configured to implement one or more of: IEEE 102.11, IEEE 102.15, IEEE 102.15.1, IEEE 102.15.2, IEEE 102.15.3, IEEE 102.15.4, IEEE 102.15.5, IEEE 102.20, and / or In some embodiments, the communication module 154 can be a hardwired data port (e.g., a USB port, a mini USB port, a USB-C connector, an HDMI port, an Ethernet port, a DisplayPort interface, a Lightning port, etc.). In some embodiments, the wireless interface 156 can also implement one or more protocols associated with cellular communication networks (e.g., GSM, CDMA, GPRS, UMTS, CDMA2000, 3GPP LTE, 4G / HSPA / WiMAX, SMS, etc.). In embodiments in which the camera system 100 is implemented as a wireless camera, the protocols implemented by the communication module 154 and the wireless interface 156 can be wireless communication protocols. The type of communication protocols implemented by the communication module 154 can vary depending on the design standards of the particular implementation.

[0050] The communication module 154 and / or the wireless interface 156 can be configured to generate a broadcast signal as an output from the camera system 100. The broadcast signal can transmit video data, parallax data, and / or control signals to an external device. For example, the broadcast signal can be transmitted to a cloud storage service (e.g., a storage service capable of scaling on demand). In some embodiments, the communication module 154 can not transmit data until the processor / SoC 102 has performed video analysis to determine that an object is in the field of view of the camera system 100.

[0051] In some embodiments, the communication module 154 can be configured to generate a manual control signal. The manual control signal can be generated in response to a signal received by the communication module 154 from a user. The manual control signal can be configured to activate the processor / SoC 102. The processor / SoC 102 can be activated in response to the manual control signal regardless of the power state of the camera system 100.

[0052] In some embodiments, the communication module 154 and / or the wireless interface 156 can be configured to receive a feature set. The received feature set can be used to detect events and / or objects. For example, the feature set can be used to perform computer vision operations. The feature set information can include instructions for the processor 102 for determining which types of objects correspond to objects and / or events of interest.

[0053] The processor 158 can be implemented using general processor circuitry. The processor 158 can be operable to interact with the video processing circuitry 102 and the memory 150 to perform various processing tasks. The processor 158 can be configured to execute computer-readable instructions. In an example, the computer-readable instructions can be stored by the memory 150. In some embodiments, the computer-readable instructions can include controller operations. Generally, input from the sensors 164 and / or human interface devices 166 are shown as being received by the processor 102. In some embodiments, the general processor 158 can be configured to receive and / or analyze data from the sensors 164 and / or HID 166 and make decisions in response to the input. In some embodiments, the processor 158 can send and / or receive data to / from other components of the camera system 100 (e.g., the battery 152, the communication module 154, and / or the wireless interface 156). Which functions of the camera system 100 are performed by the processor 102 and the general processor 158 can vary depending on the design criteria for a particular implementation.

[0054] The lens 160 can be attached to the capture device 104. The capture device 104 can be configured to receive an input signal (e.g., LIN) via the lens 160. The signal LIN can be an optical input (e.g., an analog image). The lens 160 can be implemented as an optical lens. The lens 160 can provide zoom features and / or focus features. In an example, the capture device 104 and / or the lens 160 can be implemented as a single lens assembly. In another example, the lens 160 can be implemented separately from the capture device 104.

[0055] The capture device 104 can be configured to convert the input light LIN into computer readable data. The capture device 104 can capture data received through the lens 160 to generate raw pixel data. In some embodiments, the capture device 104 can capture data received through the lens 160 to generate a bitstream (e.g., to generate a video frame). For example, the capture device 104 can receive focused light from the lens 160. The lens 160 can be oriented, tilted, panned, zoomed, and / or rotated to provide a target view from the camera system 100 (e.g., a view of a video frame, a view of a panoramic video frame captured using multiple camera systems 100a-100n, target image and reference image views for stereo vision, etc.). The capture device 104 can generate a signal (e.g., a video). The signal VIDEO can be pixel data (e.g., a sequence of pixels that can be used to generate a video frame). In some embodiments, the signal VIDEO can be video data (e.g., a sequence of video frames). The signal VIDEO can be presented to one of the inputs of the processor 102. In some embodiments, the pixel data generated by the capture device 104 can be uncompressed and / or raw data generated in response to focused light from the lens 160. In some embodiments, the output of the capture device 104 can be a digital video signal.

[0056] In an example, the capture device 104 can include a block (or circuit) 180, a block (or circuit) 182, and a block (or circuit) 184. The circuit 180 can be an image sensor. The circuit 182 can be a processor and / or logic unit. The circuit 184 can be a memory circuit (e.g., a frame buffer). The lens 160 (e.g., a camera lens) can be directed to provide a view of an environment surrounding the camera system 100. The lens 160 can be intended to capture environmental data (e.g., the light input LIN). The lens 160 can be a wide-angle lens and / or a fisheye lens (e.g., a lens capable of capturing a wide field of view). The lens 160 can be configured to capture and / or focus light for the capture device 104. Typically, the image sensor 180 is located behind the lens 160. Based on the light captured from the lens 160, the capture device 104 can generate a bitstream and / or video data (e.g., the signal VIDEO).

[0057] The capture device 104 can be configured to capture video image data (e.g., light collected and focused by the lens 160). The capture device 104 can capture data received through the lens 160 to generate a video bitstream (e.g., pixel data of a sequence of video frames). In various embodiments, the lens 160 can be implemented as a fixed focus lens. Fixed focus lenses are generally advantageous for smaller size and lower power. In an example, fixed focus lenses can be used for battery-powered, doorbell, and other low-power camera applications. In some embodiments, the lens 160 can be oriented, tilted, panned, zoomed, and / or rotated to capture an environment around the camera system 100 (e.g., capture data from a field of view). In an example, professional camera models can use an active lens system for enhanced functionality, remote control, etc.

[0058] The capture device 104 can convert received light into a digital data stream. In some embodiments, the capture device 104 can perform analog-to-digital conversion. For example, the image sensor 180 can perform photoelectric conversion on light received by the lens 160. The processor / logic 182 can convert the digital data stream into a video data stream (or bitstream), a video file, and / or a plurality of video frames. In an example, the capture device 104 can present video data as a digital video signal (e.g., VIDEO). The digital video signal can include video frames (e.g., successive digital images and / or audio). In some embodiments, the capture device 104 can include a microphone for capturing audio. In some embodiments, the microphone can be implemented as a separate component (e.g., one of the sensors 164).

[0059] Video data captured by the capture device 104 can be represented as a signal / bitstream / data VIDEO (e.g., a digital video signal). The capture device 104 can present the signal VIDEO to the processor / SoC 102. The signal VIDEO can represent video frames / video data. The signal VIDEO can be a video stream captured by the capture device 104. In some embodiments, the signal VIDEO can include pixel data operable by the processor 102 (e.g., a video processing pipeline, an image signal processor (ISP), etc.). The processor 102 can generate video frames in response to pixel data in the signal VIDEO.

[0060] The signal VIDEO can include pixel data arranged as video frames. The signal VIDEO can be an image that includes a background (e.g., captured objects and / or environment) and a speckle pattern generated by the structured light projector 106. The signal VIDEO can include a single-channel source image. The single-channel source image can be generated in response to capturing pixel data using the monocular lens 160.

[0061] Image sensor 180 can receive input light LIN from lens 160 and convert light LIN to digital data (e.g., a bitstream). For example, image sensor 180 can perform photoelectric conversion on light from lens 160. In some embodiments, image sensor 180 can have additional margins that are not used as part of the image output. In some embodiments, image sensor 180 can not have additional margins. In various embodiments, image sensor 180 can be configured to generate an RGB-IR video signal. In a field of view illuminated by infrared light only, image sensor 180 can generate a monochrome (B / W) video signal. In a field of view illuminated by both IR light and visible light, image sensor 180 can be configured to generate color information in addition to a monochrome video signal. In various embodiments, image sensor 180 can be configured to generate a video signal in response to visible light and / or infrared (IR) light.

[0062] In some embodiments, camera sensor 180 can include a rolling shutter sensor or a global shutter sensor. In an example, rolling shutter sensor 180 can implement an RGB-IR sensor. In some embodiments, capture device 104 can include a rolling shutter IR sensor and an RGB sensor (e.g., implemented as separate components). In an example, rolling shutter sensor 180 can be implemented as an RGB-IR rolling shutter complementary metal-oxide-semiconductor (CMOS) image sensor. In one example, rolling shutter sensor 180 can be configured to assert a signal indicative of a first row exposure time. In one example, rolling shutter sensor 180 can apply a mask to a monochrome sensor. In an example, the mask can include a plurality of cells containing one red pixel, one green pixel, one blue pixel, and one IR pixel. The IR pixel can contain red, green, and blue filter materials that can efficiently absorb all light in the visible spectrum while allowing longer infrared wavelengths to pass through with minimal loss. In the case of a rolling shutter, as each row (or line) of the sensor begins exposure, all pixels in that row (or line) can begin exposure at the same time.

[0063] The processor / logic unit 182 can convert the bitstream into human- visible content (e.g., video data that a normal person can understand, such as video frames and / or pixel data that can be converted into video frames by the processor 102, regardless of image quality). For example, the processor / logic unit 182 can receive raw (e.g., original) data from the image sensor 180 and generate (e.g., encode) video data (e.g., a bitstream) based on the original data. The capture device 104 can have a memory 184 to store the original data and / or the processed bitstream. For example, the capture device 104 can implement a frame memory and / or buffer 184 to store (e.g., provide temporary storage and / or caching) one or more video frames (e.g., digital video signals). In some embodiments, the processor / logic unit 182 can perform analysis and / or correction on video frames stored in the memory / buffer 184 of the capture device 104. The processor / logic unit 182 can provide status information regarding the captured video frames.

[0064] The structured light projector 106 can include a block (or circuit) 186. The circuit 186 can implement a structured light source. The structured light source 186 can be configured to generate a signal (e.g., SLP). The signal SLP can be a structured light pattern (e.g., a speckle pattern). The signal SLP can be projected onto an environment near the camera system 100. The structured light pattern SLP can be captured by the capture device 104 as part of the light input LIN.

[0065] The structured light pattern lens 162 can be a lens for the structured light projector 106. The structured light pattern lens 162 can be configured to enable the structured light SLP produced by the structured light source 186 of the structured light projector 106 to be emitted while protecting the structured light source 186. The structured light pattern lens 162 can be configured to break up the laser pattern produced by the structured light source 186 into a pattern array (e.g., a dense dot pattern array for a speckle pattern).

[0066] In an example, the structured light source 186 can be implemented as an array of vertical cavity surface emitting lasers (VCSELs) and a lens. However, other types of structured light sources can be implemented to meet design criteria for a particular application. In an example, the array of VCSELs is generally configured to generate a laser pattern (e.g., the signal SLP). The lens is generally configured to break up the laser pattern into a dense dot pattern array. In an example, the structured light source 186 can implement a near-infrared (NIR) light source. In various embodiments, the light source of the structured light source 186 can be configured to emit light at a wavelength of approximately 940 nanometers (nm), which is not visible to the human eye. However, other wavelengths can be utilized. In an example, wavelengths in the range of approximately 800 nm - 1000 nm can be utilized.

[0067] The sensor 164 can implement multiple sensors including, but not limited to, a motion sensor, an ambient light sensor, a proximity sensor (e.g., ultrasonic, radar, lidar, etc.), an audio sensor (e.g., microphone), etc. In embodiments implementing a motion sensor, the sensor 164 can be configured to detect motion anywhere in the field of view monitored by the camera system 100 (or some locations outside the field of view). In various embodiments, detection of motion can be used as one threshold for activating the capture device 104. The sensor 164 can be implemented as an internal component of the camera system 100 and / or a component external to the camera system 100. In an example, the sensor 164 can be implemented as a passive infrared (PIR) sensor. In another example, the sensor 164 can be implemented as a smart motion sensor. In yet another example, the sensor 164 can be implemented as a microphone. In embodiments implementing a smart motion sensor, the sensor 164 can include a low resolution image sensor configured to detect motion and / or people.

[0068] In various embodiments, the sensor 164 can generate a signal (e.g., SENS). The signal SENS can include various data (or information) collected by the sensor 164. In an example, the signal SENS can include data collected in response to motion detected in the monitored field of view, ambient light levels in the monitored field of view, and / or sounds picked up in the monitored field of view. However, other types of data can be collected and / or generated based on design criteria of a particular application. The signal SENS can be presented to the processor / SoC 102. In an example, the sensor 164 can generate (assert) the signal SENS when motion is detected in the field of view monitored by the camera system 100. In another example, the sensor 164 can generate (assert) the signal SENS when audio triggers in the field of view monitored by the camera system 100. In yet another example, the sensor 164 can be configured to provide directional information about motion and / or sounds detected in the field of view. The directional information can also be communicated to the processor / SoC 102 via the signal SENS.

[0069] The HID 166 can implement an input device. For example, the HID 166 can be configured to receive human input. In one example, the HID 166 can be configured to receive a password input from a user. In some embodiments, the camera system 100 can include a keyboard, a touchpad (or screen), a doorbell switch, and / or other human interface device (HID) 166. In an example, the sensor 164 can be configured to determine when an object approaches the HID 166. In examples where the camera system 100 is implemented as part of an access control application, the capture device 104 can be turned on to provide images for recognizing a person attempting access, and lighting can be turned on for the touchpad 166. For example, a combination of input from the HID 166 (e.g., a password or PIN code) can be combined with liveliness determination and / or depth analysis performed by the processor 102 to implement two-factor authentication.

[0070] The processor / SoC 102 can receive a signal VIDEO and a signal SENS. The processor / SoC 102 can generate one or more video output signals (e.g., VIDOUT), one or more control signals (e.g., CTRL), and / or one or more depth data signals (e.g., DIMAGES) based on the signal VIDEO, the signal SENS, and / or other inputs. In some embodiments, the signals VIDOUT, DIMAGES, and CTRL can be generated based on analysis of the signal VIDEO and / or objects detected in the signal VIDEO.

[0071] In various embodiments, the processor / SoC 102 can be configured to perform one or more of feature extraction, object detection, object tracking, 3D reconstruction, and object recognition. For example, the processor / SoC 102 can determine motion information and / or depth information by analyzing frames from the signal VIDEO and comparing the frames to previous frames. The comparison can be used to perform digital motion estimation. In some embodiments, the processor / SoC 102 can be configured to generate a video output signal VIDOUT including video data and / or a depth data signal DIMAGES including a disparity map and a depth map from the signal VIDEO. The video output signal VIDOUT and / or the depth data signal DIMAGES can be presented to the memory 150, the communication module 154, and / or the wireless interface 156. In some embodiments, the video signal VIDOUT and / or the depth data signal DIMAGES can be used internally by the processor 102 (e.g., not presented as an output).

[0072] The signal VIDOUT can be presented to the communication device 156. In some embodiments, the signal VIDOUT can include encoded video frames generated by the processor 102. In some embodiments, the encoded video frames can include a complete video stream (e.g., encoded video frames representing all video captured by the capture device 104). The encoded video frames can be an encoded, cropped, stitched, and / or enhanced version of the pixel data received from the signal VIDEO. In an example, the encoded video frames can be a high resolution, digital, encoded, de-warp, stabilized, cropped, blended, stitched, and / or rolling shutter effect corrected version of the signal VIDEO.

[0073] In some embodiments, the signal VIDOUT can be generated based on video analysis (e.g., computer vision operations) performed by the processor 102 on the generated video frames. The processor 102 can be configured to perform computer vision operations to detect objects and / or events in the video frames and then convert the detected objects and / or events into statistics and / or parameters. In one example, the data determined by the computer vision operations can be converted by the processor 102 into a human readable format. The data from the computer vision operations can be used to detect objects and / or events. The computer vision operations can be performed locally by the processor 102 (e.g., without needing to communicate with an external device to offload the computational operations). For example, the locally performed computer vision operations can enable the computer vision operations to be performed by the processor 102 and avoid heavy video processing running on a backend server. Avoiding running video processing on a backend (e.g., remotely located) server can protect privacy.

[0074] In some embodiments, the signal VIDOUT can be data (e.g., video analysis results, audio / speech analysis results, etc.) generated by the processor 102 that can be communicated to a cloud computing service in order to aggregate information and / or provide training data for machine learning (e.g., to improve object detection, to improve audio detection, etc.). In some embodiments, the signal VIDOUT can be provided to a cloud service for bulk storage (e.g., to enable a user to retrieve the encoded video using a smartphone and / or a desktop computer). In some embodiments, the signal VIDOUT can include data extracted from the video frames (e.g., results of computer vision) and can communicate the results to another device (e.g., a remote server, a cloud computing system, etc.) to offload analysis of the results to another device (e.g., offload analysis of the results to a cloud computing service rather than performing all analysis locally). The type of information communicated by the signal VIDOUT can vary depending on the design criteria of the particular implementation.

[0075] The signal CTRL can be configured to provide a control signal. The signal CTRL can be generated in response to a decision made by the processor 102. In an example, the signal CTRL can be generated in response to a detected object and / or a feature extracted from a video frame. The signal CTRL can be configured to enable, disable, change a mode of operation of another device. In one example, a door controlled by an electronic lock can be locked / unlocked in response to the signal CTRL. In another example, a device can be set to a sleep mode (e.g., a low power mode) and / or activated from a sleep mode in response to the signal CTRL. In yet another example, an alarm and / or notification can be generated in response to the signal CTRL. The type of device controlled by the signal CTRL and / or the reaction performed by the device in response to the signal CTRL can vary according to design criteria of a particular implementation.

[0076] The signal CTRL can be generated based on data received by the sensor 164 (e.g., a temperature reading, a motion sensor reading, etc.). The signal CTRL can be generated based on input from the HID 166. The signal CTRL can be generated based on a detected behavior of a person in a video frame by the processor 102. The signal CTRL can be generated based on a type of detected object (e.g., a person, an animal, a vehicle, etc.). The signal CTRL can be generated in response to detecting a particular type of object at a particular location. The processor 102 can be configured to generate the signal CTRL in response to a sensor fusion operation (e.g., aggregating information received from different sources). The conditions for generating the signal CTRL can vary according to design criteria of a particular implementation.

[0077] The signal DIMAGES can include one or more of a depth map and / or a disparity map generated by the processor 102. The signal DIMAGES can be generated in response to a 3D reconstruction performed on monocular single channel images. The signal DIMAGES can be generated in response to an analysis of captured video data and a structured light pattern SLP.

[0078] A multi-step approach for activating and / or disabling the capture device 104 based on the output of the motion sensor 164 and / or any other power consumption features of the camera system 100 can be implemented to reduce the power consumption of the camera system 100 and extend the life of the battery 152. The motion sensor in the sensor 164 can have a low consumption (e.g., less than 10 W) on the battery 152. In an example, the motion sensor of the sensor 164 can be configured to remain on (e.g., always active) unless disabled in response to feedback from the processor / SoC 102. The video analysis performed by the processor / SoC 102 can have a relatively larger consumption (e.g., greater than the motion sensor 164) on the battery 152. In an example, the processor / SoC 102 can be in a low power state (or powered off) until some motion is detected by the motion sensor of the sensor 164.

[0079] The camera system 100 can be configured to operate using various power states. For example, in a powered off state (e.g., sleep state, low power state), the motion sensor of the sensor 164 and the processor / SoC 102 can be on, and other components of the camera system 100 (e.g., the image capture device 104, the memory 150, the communication module 154, etc.) can be off. In another example, the camera system 100 can operate in an intermediate state. In the intermediate state, the image capture device 104 can be on, and the memory 150 and / or the communication module 154 can be off. In yet another example, the camera system 100 can operate in a powered on (or high power) state. In the powered on state, the sensor 164, the processor / SoC 102, the capture device 104, the memory 150, and / or the communication module 154 can be on. The camera system 100 can consume some power (e.g., a relatively small and / or minimal amount of power) from the battery 152 in the powered off state. In the powered on state, the camera system 100 can consume more power from the battery 152. The number of power states and / or the number of components of the camera system 100 that are on when the camera system 100 operates in each power state can vary depending on the design criteria of the particular implementation.

[0080] In some embodiments, the camera system 100 can be implemented as a system on a chip (SoC). For example, the camera system 100 can be implemented as a printed circuit board that includes one or more components. The camera system 100 can be configured to perform intelligent video analysis on video frames of a video. The camera system 100 can be configured to crop and / or enhance the video.

[0081] In some embodiments, the video frames can be of a certain view (or a derivative of a certain view) captured by the capture device 104. The pixel data signals can be enhanced (e.g., color conversion, noise filtering, auto exposure, auto white balance, auto focus, etc.) by the processor 102. In some embodiments, the video frames can provide a series of cropped and / or enhanced video frames that improve the view from the perspective of the camera system 100 (e.g., provide night vision, provide high dynamic range (HDR) imaging, provide more viewing area, highlight detected objects, provide additional data (e.g., digital distance to detected objects), etc.) to enable the processor 102 to see a location better than a person can see with human vision.

[0082] The encoded video frames can be processed locally. In one example, the encoded video can be stored locally by the memory 150 to enable the processor 102 to facilitate computer vision analysis internally (e.g., without first uploading the video frames to a cloud service). The processor 102 can be configured to select video frames to be packaged as a video stream that can be sent over a network (e.g., a bandwidth limited network).

[0083] In some embodiments, the processor 102 can be configured to perform a sensor fusion operation. The sensor fusion operation performed by the processor 102 can be configured to analyze information from multiple sources (e.g., the capture device 104, the sensors 164, and the HID 166). By analyzing various data from different sources, the sensor fusion operation can be able to make inferences about the data that can not be possible from just one of the data sources. For example, the sensor fusion operation implemented by the processor 102 can analyze video data (e.g., mouth movement of a person) as well as speech patterns from directional audio. The different sources can be used to develop a model of the scene to support decision making. For example, the processor 102 can be configured to compare the synchronization of detected speech patterns with mouth movement in video frames to determine which person in the video frames is speaking. The sensor fusion operation can also provide temporal, spatial, and / or reliability correlations among the data being received.

[0084] In some embodiments, the processor 102 can implement convolutional neural network capabilities. The convolutional neural network capabilities can use deep learning techniques to implement computer vision. The convolutional neural network capabilities can be configured to implement pattern and / or image recognition using a training process through multiple layers of feature detection. The computer vision and / or convolutional neural network capabilities can be performed locally by the processor 102. In some embodiments, the processor 102 can receive training data and / or feature set information from an external source. For example, an external device (e.g., a cloud service) can have access to various data sources to use as training data that can not be available to the camera system 100. However, computer vision operations performed using the feature set can be performed using the computing resources of the processor 102 within the camera system 100.

[0085] The video pipeline of the processor 102 can be configured to perform de-warping, cropping, enhancement, rolling shutter correction, stabilization, downscaling, packing, compression, conversion, blending, synchronization, and / or other video operations locally. The video pipeline of the processor 102 can enable multi-stream support (e.g., generating multiple bitstreams in parallel, each including a different bit rate). In an example, the video pipeline of the processor 102 can implement an image signal processor (ISP) with a 320M pixel / second input pixel rate. The architecture of the video pipeline of the processor 102 can enable high resolution video and / or high bit rate video data to be performed in real-time and / or near real-time. The video pipeline of the processor 102 can implement computer vision processing, stereo vision processing, object detection, 3D denoising, fisheye lens correction (e.g., real-time 360 degree de-warping and lens distortion correction), oversampling, and / or high dynamic range processing on 4K resolution video data. In one example, the architecture of the video pipeline can implement 4K ultra-high resolution with H.264 encoding at double real-time speed (e.g., 60 fps), 4K ultra-high resolution with H.265 / HEVC and / or 4K AVC encoding at 30 fps (e.g., 4K P30 AVC and HEVC encoding with multi-stream support). The types of video operations and / or the types of video data operated on by the processor 102 can vary depending on the design criteria of the particular implementation.

[0086] The camera sensor 180 can implement a high resolution sensor. Using the high resolution sensor 180, the processor 102 can combine oversampling of the image sensor 180 with digital zoom within the cropped area. Each of the oversampling and the digital zoom can be one of the video operations performed by the processor 102. The oversampling and the digital zoom can be implemented to provide a higher resolution image within the total size limitations of the cropped area.

[0087] In some embodiments, the lens 160 can implement a fisheye lens. One of the video operations implemented by the processor 102 can be a de-warping operation. The processor 102 can be configured to de-warp the generated video frames. The de-warping can be configured to reduce and / or remove severe distortions caused by the fisheye lens and / or other lens characteristics. For example, the de-warping can reduce and / or eliminate the convex effect to provide a straight-line image.

[0088] The processor 102 can be configured to crop (e.g., trim) a region of interest from the full video frame (e.g., generate a region of interest video frame). The processor 102 can generate the video frame and select the region. In an example, cropping the region of interest can generate a second image. The cropped image (e.g., the region of interest video frame) can be smaller than the original video frame (e.g., the cropped image can be a portion of the captured video).

[0089] The region of interest can be dynamically adjusted based on the location of the audio source. For example, the detected audio source can be moving and the location of the detected audio source can move with the capture of the video frames. The processor 102 can update the selected region of interest coordinates and dynamically update the cropped portion (e.g., an implementation of a directional microphone as one or more of the sensors 164 can dynamically update the location based on the captured directional audio). The cropped portion can correspond to the selected region of interest. The cropped portion can change as the region of interest changes. For example, the selected coordinates of the region of interest can change from frame to frame and the processor 102 can be configured to crop the selected region in each frame.

[0090] The processor 102 can be configured to oversample the image sensor 180. The oversampling of the image sensor 180 can result in a higher resolution image. The processor 102 can be configured to digitally magnify a region of the video frame. For example, the processor 102 can digitally magnify the cropped region of interest. For example, the processor 102 can establish a region of interest based on the directional audio, crop the region of interest, and then digitally magnify the cropped region of interest video frame.

[0091] The de-warping operation performed by the processor 102 can adjust the visual content of the video data. The adjustment performed by the processor 102 can make the visual content appear natural (e.g., appear as if seen by a person viewing a location corresponding to the field of view of the capture device 104). In an example, the de-warping can change the video data to generate straight-line video frames (e.g., correct artifacts caused by the lens characteristics of the lens 160). The de-warping operation can be implemented to correct distortions caused by the lens 160. The adjusted visual content can be generated to enable more accurate and / or reliable object detection.

[0092] Various features (e.g., de-warping, digital zoom, cropping, etc.) can be implemented in the processor 102 as hardware modules. Implementing hardware modules can increase the video processing speed of the processor 102 (e.g., faster than software implementations). Hardware implementations can enable video to be processed while reducing the amount of latency. The hardware components used can vary depending on the design standards of the particular implementation.

[0093] The processor 102 is shown to include a number of blocks (or circuits) 190a-190n. The blocks 190a-190n can implement various hardware modules implemented by the processor 102. The hardware modules 190a-190n can be configured to provide various hardware components to implement a video processing pipeline. The circuits 190a-190n can be configured to receive pixel data VIDEO, generate video frames from the pixel data, perform various operations on the video frames (e.g., de-warping, rolling shutter correction, cropping, zooming, image stabilization, 3D reconstruction, etc.), prepare the video frames for communication with external hardware (e.g., encoding, packing, color correction, etc.), resolve feature sets, implement various operations for computer vision (e.g., object detection, segmentation, classification, etc.), etc. The hardware modules 190a-190n can be configured to implement various security features (e.g., secure boot, I / O virtualization, etc.). Various implementations of the processor 102 can not necessarily utilize all of the features of the hardware modules 190a-190n. The features and / or functionality of the hardware modules 190a-190n can vary depending on the design standards of the particular implementation. Details of the hardware modules 190a-190n can be described in connection with U.S. Patent Application No. 16 / 831,549, filed April 16, 2020, U.S. Patent Application No. 16 / 288,922, filed February 28, 2019, U.S. Patent Application 15 / 593,493, filed May 12, 2017 (now U.S. Patent No. 10,437,600), U.S. Patent Application No. 15 / 931,942, filed May 14, 2020, U.S. Patent Application No. 16 / 991,344, filed August 12, 2020, appropriate portions of which are incorporated by reference in their entirety.

[0094] The hardware modules 190a-190n can be implemented as specialized hardware modules. In contrast to software implementations, using specialized hardware modules 190a-190n to implement various functions of the processor 102 can enable the processor 102 to be highly optimized and / or customized to limit power consumption, reduce heat generation, and / or increase processing speed. The hardware modules 190a-190n can be customizable and / or programmable to implement multiple types of operations. Implementing specialized hardware modules 190a-190n can enable the hardware used to perform each type of computation to be optimized for speed and / or efficiency. For example, the hardware modules 190a-190n can implement multiple relatively simple operations that are frequently used in computer vision operations, which together can enable computer vision operations to be performed in real-time. The video pipeline can be configured to identify objects. Objects can be identified by interpreting numerical and / or symbolic information to determine that the visual data represents a particular type of object and / or feature. For example, the number of pixels and / or the color of pixels of the video data can be used to identify some portions of the video data as objects. The hardware modules 190a-190n can enable computationally intensive operations (e.g., computer vision operations, video encoding, video transcoding, 3D reconstruction, etc.) to be performed locally by the camera system 100.

[0095] One of the hardware modules 190a-190n (e.g., 190a) can implement a scheduler circuit. The scheduler circuit 190a can be configured to store a directed acyclic graph (DAG). In an example, the scheduler circuit 190a can be configured to generate and store the directed acyclic graph in response to receiving (e.g., loading) the feature set information. The directed acyclic graph can define video operations to be performed to extract data from video frames. For example, the directed acyclic graph can define various mathematical weighting (e.g., neural network weights and / or biases) to be applied in performing computer vision operations to classify various groups of pixels as particular objects.

[0096] The scheduler circuit 190a can be configured to parse the acyclic graph to generate various operators. The operators can be scheduled by the scheduler circuit 190a in one or more of the other hardware modules 190a-190n. For example, one or more of the hardware modules 190a-190n can implement a hardware engine configured to perform a particular task (e.g., a hardware engine designed to perform a particular mathematical operation that is repeatedly used to perform computer vision operations). The scheduler circuit 190a can schedule the operators based on when the operators are ready to be processed by the hardware engines 190a-190n.

[0097] The scheduler circuit 190a can time-multiplex tasks to the hardware modules 190a-190n based on availability of the hardware modules 190a-190n to perform work. The scheduler circuit 190a can resolve the directed acyclic graph into one or more data flows. Each data flow can include one or more operators. Once the directed acyclic graph is resolved, the scheduler circuit 190a can assign the data flows / operators to the hardware engines 190a-190n and send the relevant operator configuration information to launch the operators.

[0098] Each directed acyclic graph Figure Two The binary representation can be an ordered traversal of the directed acyclic graph, with descriptors and operators interleaved based on data dependencies. Descriptors generally provide registers that link data buffers to specific operands in related operators. In various embodiments, an operator can not appear in the directed acyclic graph representation until all related descriptors are declared for operands.

[0099] One of the hardware modules 190a-190n (e.g., 190b) can implement a convolutional neural network (CNN) module. The CNN module 190b can be configured to perform computer vision operations on video frames. The CNN module 190b can be configured to implement recognition of objects through multi-layer feature detection. The CNN module 190b can be configured to compute descriptors based on the performed feature detection. The descriptors can enable the processor 102 to determine a likelihood that pixels of a video frame correspond to a particular object (e.g., a particular make / model / year of a vehicle, identify a person as a particular individual, detect an animal type, etc.).

[0100] The CNN module 190b can be configured to implement convolutional neural network capabilities. The CNN module 190b can be configured to implement computer vision using deep learning techniques. The CNN module 190b can be configured to implement pattern and / or image recognition using a training process through multi-layer feature detection. The CNN module 190b can be configured to perform inference for a machine learning model.

[0101] The CNN module 190b can be configured to perform feature extraction and / or matching only in hardware. Feature points generally represent regions of interest in a video frame (e.g., corners, edges, etc.). By tracking feature points over time, an estimate of the ego-motion of the capture platform or a motion model of observed objects in a scene can be generated. To track feature points, matching operations are typically consolidated in the CNN module 190b by hardware to find the most likely correspondence between feature points in a reference video frame and a target video frame. In matching pairs of reference and target feature points, each feature point can be represented by a descriptor (e.g., image patch, SIFT, BRIEF, ORB, FREAK, etc.). Implementing the CNN module 190b using dedicated hardware circuitry can enable real-time computation of descriptor matching distances.

[0102] The CNN module 190b can be configured to perform face detection, face recognition, and / or liveness determination. For example, face detection, face recognition, and / or liveness determination can be performed based on a trained neural network implemented by the CNN module 190b. In some embodiments, the CNN module 190b can be configured to generate depth images from structured light patterns. The CNN module 190b can be configured to perform various detection and / or recognition operations and / or to perform 3D recognition operations.

[0103] The CNN module 190b can be a specialized hardware module configured to perform feature detection of video frames. Features detected by the CNN module 190b can be used to compute descriptors. The CNN module 190b can determine, responsive to the descriptors, a likelihood that a pixel in a video frame belongs to a particular object and / or some characteristic of an object. For example, using descriptors, the CNN module 190b can determine a probability that a pixel corresponds to a particular object (e.g., a person, a piece of furniture, a pet, a vehicle, etc.) and / or a characteristic of an object (e.g., a shape of an eye, a distance between facial features, a vehicle hood, a body part, a vehicle license plate, a face, a piece of clothing worn by a person, etc.). Implementing the CNN module 190b as a specialized hardware module of the processor 102 can enable the apparatus 100 to perform computer vision operations locally (e.g., on-chip) without relying on processing power of remote devices (e.g., transferring data to a cloud computing service).

[0104] The computer vision operations performed by the CNN module 190b can be configured to perform feature detection on the video frames to generate descriptors. The CNN module 190b can perform object detection to determine regions in the video frames that have a high likelihood of matching a particular object. In one example, the type of object to be matched (e.g., a reference object) can be customized using an open operator stack (enabling programmability of the processor 102 to implement various artificial neural networks defined by directed acyclic graphs, each providing instructions for performing various types of object detection). The CNN module 190b can be configured to perform local masking on the regions that have a high likelihood of matching the particular object to detect the object.

[0105] In some embodiments, the CNN module 190b can determine locations (e.g., 3D coordinates and / or position coordinates) of various features (e.g., characteristics) of the detected object. In one example, 3D coordinates can be used to determine the locations of a person’s arms, legs, chest, and / or eyes. One position coordinate of the vertical position of a body part in 3D space and another coordinate of the horizontal position of the body part in 3D space can be stored. In some embodiments, the distance from the lens 160 can represent one coordinate of the depth position of the body part in 3D space (e.g., a position coordinate on a third axis). Using the locations of various body parts in 3D space, the processor 102 can determine the body position and / or body characteristics of the detected person.

[0106] The CNN module 190b can be pre-trained (e.g., configured to perform computer vision to detect objects based on training data received to train the CNN module 190b). For example, the results of the training data (e.g., a machine learning model) can be preprogrammed and / or loaded into the processor 102. The CNN module 190b can make inferences on the machine learning model (e.g., to perform object detection). The training can include determining weight values for each layer of the neural network model. For example, weight values can be determined for each layer of feature extraction (e.g., convolutional layers) and / or for classification (e.g., fully connected layers). The weight values learned by the CNN module 190b can vary depending on the design criteria of the particular implementation.

[0107] The CNN module 190b can implement feature extraction and / or object detection by performing convolution operations. The convolution operations can be hardware-accelerated for fast (e.g., real-time) computations that can be performed while consuming low power. In some embodiments, the convolution operations performed by the CNN module 190b can be used to perform computer vision operations. In some embodiments, the convolution operations performed by the CNN module 190b can be used for any functionality performed by the processor 102 that can involve computing convolution operations (e.g., 3D reconstruction).

[0108] The convolution operation can include sliding the feature detection window along the layers while performing computations (e.g., matrix operations). The feature detection window can apply filters to the pixels and / or extract features associated with each layer. The feature detection window can be applied to one pixel and multiple surrounding pixels. In an example, the layers can be represented as matrices of values representing pixels and / or features of one of the layers, and the filters applied by the feature detection window can be represented as matrices. The convolution operation can apply matrix multiplication between the regions of the current layer covered by the feature detection window. The convolution operation can slide the feature detection window along the regions of the layers to generate results representing each of the regions. The size of the regions, the type of operation to apply the filters, and / or the number of layers can vary according to design criteria of a particular implementation.

[0109] Using the convolution operation, the CNN module 190b can compute multiple features of the pixels of the input image in each extraction step. For example, each layer can receive input from a set of features in a small neighborhood (e.g., a region) located at a previous layer (e.g., a local receptive field). The convolution operation can extract basic visual features (e.g., oriented edges, end points, corners, etc.) that are then combined by higher layers. Since the feature extraction window operates on a pixel and its nearby pixels (or sub-pixels), the result of the operation can have position invariance. The layers can include convolution layers, pooling layers, non-linear layers, and / or fully connected layers. In an example, the convolution operation can learn to detect edges from raw pixels (e.g., a first layer), then use the detected edges from the previous layer to detect shapes in the next layer, and then use the shapes to detect higher level features (e.g., facial features, pets, vehicles, vehicle parts, furniture, etc.) in higher layers, and the last layer can be a classifier that uses the higher level features.

[0110] The CNN module 190b can perform a dataflow for feature extraction and matching, including two-stage detection, a warp operator, a component operator that manipulates a list of components (e.g., a component can be a region of a vector that shares a common attribute and can be combined with a bounding box), a matrix inverse operator, a dot product operator, a convolution operator, a conditional operator (e.g., multiplexing and de-multiplexing), a remap operator, a min-max-reduce operator, a pooling operator, a non-min, non-max suppression operator, a scan window based non-max suppression operator, an aggregation operator, a dispersion operator, a statistical operator, a classification operator, an integral image operator, a comparison operator, an indexing operator, a pattern matching operator, a feature extraction operator, a feature detection operator, a two-stage object detection operator, a score generation operator, a block reduction operator, and an up-sampling operator. The type of operations performed by the CNN module 190b for extracting features from training data can vary according to design criteria of a particular implementation.

[0111] Each of the hardware modules 190a-190n can implement a processing resource (or hardware resource or hardware engine). The hardware engines 190a-190n can be used to perform specific processing tasks. In some configurations, the hardware engines 190a-190n can operate in parallel and independently of one another. In other configurations, the hardware engines 190a-190n can operate in conjunction with one another to perform assigned tasks. One or more of the hardware engines 190a-190n can be homogeneous processing resources (all of the circuits 190a-190n can have the same capabilities) or heterogeneous processing resources (two or more of the circuits 190a-190n can have different capabilities).

[0112] Referring to Figure 4 , Figure 4 A diagram illustrating processing circuitry of a camera system implementing convolutional neural networks and matrix computations configured to perform high resolution 2D image and depth map image 3D reconstruction and accelerated alignment using low-bitwidth floating point representations is shown. In an example, the processing circuitry of the camera system 100 can be configured for various applications including, but not limited to, autonomous and semi-autonomous vehicles (e.g., cars, trucks, motorcycles, agricultural machinery, drones, airplanes, etc.), manufacturing, and / or security and surveillance systems. In contrast to general purpose computers, the processing circuitry of the camera system 100 typically includes hardware circuitry optimized to provide high performance image processing and computer vision pipelines with minimal area and utilizing minimal power consumption. In an example, various operations for performing image processing, feature detection / extraction, 3D reconstruction, and / or object detection / classification for computer (or machine) vision can be implemented using hardware modules designed to reduce computational complexity and efficient use of resources.

[0113] In an example embodiment, the processing circuitry 100 can include a processor 102, a memory 150, a general purpose processor 158, and / or a memory bus 200. The general purpose processor 158 can implement a first processor. The processor 102 can implement a second processor. In an example, the circuit 102 can implement a computer vision processor. In an example, the processor 102 can be an intelligent vision processor. The memory 150 can implement an external memory (e.g., a memory external to the circuit 158 and 102). In an example, the circuit 150 can be implemented as a dynamic random access memory (DRAM) circuit. The processing circuitry of the camera system 100 can include other components (not shown). The number, type, and / or arrangement of components of the processing circuitry of the camera system 100 can vary depending on design criteria of a particular implementation.

[0114] The general purpose processor 158 can be operative to interact with the circuit 102 and the circuit 150 to perform various processing tasks. In an example, the processor 158 can be configured as a controller of the circuit 102. The processor 158 can be configured to execute computer readable instructions. In one example, the computer readable instructions can be stored by the circuit 150. In some embodiments, the computer readable instructions can include controller operations. The processor 158 can be configured to communicate with the circuit 102 and / or access results produced by components of the circuit 102. In an example, the processor 158 can be configured to utilize the circuit 102 to perform operations associated with one or more neural network models.

[0115] In an example, the processor 102 generally includes a scheduler circuit 190a, a block (or circuit) 202, one or more blocks (or circuits) 204a-204n, a block (or circuit) 206, and a path 208. The block 202 can implement a directed acyclic graph (DAG) memory. The DAG memory 202 can include a CNN module 190b and / or weight / bias values 210. The blocks 204a-204n can implement hardware resources (or engines). The block 206 can implement a shared memory circuit. In example embodiments, one or more of the circuits 204a-204n can include blocks (or circuits) 212a-212n. In the illustrated example, the circuit 212a and the circuit 212b are implemented as representative examples of respective hardware engines 204a-204b. One or more of the circuit 202, the circuits 204a-204n, and / or the circuit 206 can be example implementations of the hardware modules 190a-190n shown in association therewith. Figure 3

[0116] In an example, the processor 158 can be configured to program the circuit 102 with one or more pre-trained artificial neural network models (ANN) including a convolutional neural network (CNN) 190b having a plurality of output frames according to embodiments of the present application and weight / kernel (WGTS) 210 used by the CNN module 190b. In various embodiments, the CNN module 190b can be configured (trained) for operation in an edge device. In an example, processing circuitry of the camera system 100 can be coupled to a sensor (e.g., a video camera, etc.) configured to generate data input. The processing circuitry of the camera system 100 can be configured to generate one or more outputs responsive to the data input from the sensor based on one or more inferences made by executing the pre-trained CNN module 190b using the weight / kernel (WGTS) 210. The operations performed by the processor 158 can vary according to design criteria of a particular implementation.

[0117] ​In various embodiments, the circuit 150 can implement a dynamic random access memory (DRAM) circuit. The circuit 150 is generally operable to store a multi-dimensional array of input data elements and various forms of output data elements. The circuit 150 can exchange input data elements and output data elements with the processor 158 and the processor 102.

[0118] The processor 102 can implement a computer vision processor circuit. In an example, the processor 102 can be configured to implement various functions for computer vision. The processor 102 is generally operable to perform specific processing tasks arranged by the processor 158. In various embodiments, all or portions of the processor 102 can be implemented separately in hardware. The processor 102 can directly perform data flow involving execution of the CNN module 190b and generated by software (e.g., directed acyclic graph, etc.) that specifies processing (e.g., computer vision, 3D reconstruction, etc.) tasks. In some embodiments, the processor 102 can be a representative example of numerous computer vision processors implemented by the processing circuit of the camera system 100 and configured to operate together.

[0119] In an example, the circuit 212a can implement a convolution operation. In another example, the circuit 212b can be configured to provide a dot product operation. The convolution and dot product operations can be used to perform computer (or machine) vision tasks (e.g., as part of an object detection process, etc.). In yet another example, one or more of the circuits 204c-204n can include blocks (or circuits) 212c-212n (not shown) for providing multi-dimensional convolution calculations. In yet another example, one or more of the circuits 204a-204n can be configured to perform 3D reconstruction tasks.

[0120] In an example, the circuit 102 can be configured to receive a directed acyclic graph (DAG) from the processor 158. The DAG received from the processor 158 can be stored in the DAG memory 202. The circuit 102 can be configured to execute the DAG for the CNN module 190b using the circuits 190a, 204a-204n, and 206.

[0121] A plurality of signals (e.g., OP_A-OP_N) can be exchanged between the circuit 190a and the respective circuits 204a-204n. Each of the signals OP_A-OP_N can convey execution operation information and / or yield operation information. A plurality of signals (e.g., MEM_A-MEM_N) can be exchanged between the various circuits 204a-204n and the circuit 206. The signals MEM_A-MEM_N can carry data. A signal (e.g., DRAM) can be exchanged between the circuit 150 and the circuit 206. The signal DRAM can transfer data between the circuit 150 and 190a (e.g., on the transfer path 208).

[0122] Circuitry 190a can implement a scheduler circuit. The scheduler circuit 190a can generally be operable to schedule tasks among circuitry 204a-204n to perform various computer vision related tasks defined by processor 158. Individual tasks can be assigned to circuitry 204a-204n by the scheduler circuit 190a. The scheduler circuit 190a can assign individual tasks in response to parsing a directed acyclic graph (DAG) provided by processor 158. The scheduler circuit 190a can time multiplex tasks to circuitry 204a-204n based on availability of circuitry 204a-204n to perform work.

[0123] Each circuit 204a-204n can implement a processing resource (or hardware engine). The hardware engine 204a-204n can generally be operable to perform a specific processing task. The hardware engine 204a-204n can be implemented to include specialized hardware circuitry that is optimized for high performance and low power consumption while performing a specific processing task. In some configurations, the hardware engine 204a-204n can operate in parallel and independently of one another. In other configurations, the hardware engine 204a-204n can operate collectively between one another to perform assigned tasks.

[0124] The hardware engine 204a-204n can be homogeneous processing resources (e.g., all circuitry 204a-204n can have the same capabilities) or heterogeneous processing resources (e.g., two or more circuitry 204a-204n can have different capabilities). The hardware engine 204a-204n are generally configured to perform operators, which can include, but are not limited to, a resample operator, a warp operator, a component operator that manipulates a list of components (e.g., components can be vector regions that share a common attribute and can be grouped together with a bounding box), a matrix inverse operator, a dot product operator, a convolution operator, a conditional operator (e.g., mux and de-mux), a remap operator, a min-max-reduce operator, a pooling operator, a non-min, non-max suppression operator, an aggregate operator, a scatter operator, a statistics operator, a classification operator, an integral image operator, an upsample operator, and a power of two downsample operator, among others.

[0125] In an example, the hardware engines 204a-204n can include matrices stored in various memory buffers. The matrices stored in the memory buffers can enable initialization of a convolution operator. The convolution operator can be configured to efficiently perform calculations that are repeatedly performed on a convolution function. In an example, the hardware engines 204a-204n implementing the convolution operator can include a plurality of mathematical circuits configured to process multi-bit input values and operate in parallel. The convolution operator can provide an efficient and versatile solution for computer vision and / or 3D reconstruction by computing a convolution (also referred to as cross-correlation) using one or higher dimensional kernels. Convolution can be used for computer vision operations such as object detection, object recognition, edge enhancement, image smoothing, etc. The techniques and / or architecture implemented by the present disclosure are operable to compute a convolution of an input array with a kernel. Details of the convolution operator can be described in relation to U.S. Patent No. 10,310,768, filed January 11, 2017, which is incorporated by reference herein in its entirety as to its pertinent portions.

[0126] In various embodiments, the hardware engines 204a-204n can be implemented individually as hardware circuits. In some embodiments, the hardware engines 204a-204n can be implemented as general purpose engines that can be configured to operate as special purpose machines (or engines) through circuit customization and / or software / firmware. In some embodiments, the hardware engines 204a-204n can alternatively be implemented as one or more instances or threads of program code executing on the processor 158 and / or one or more processors 102, including but not limited to a vector processor, a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). In some embodiments, one or more of the hardware engines 204a-204n can be selected for a particular process and / or thread by the scheduler 190a. The scheduler 190a can be configured to assign the hardware engines 204a-204n to particular tasks in response to parsing the directed acyclic graphs stored in the DAG memory 202.

[0127] Circuit 206 can implement a shared memory circuit. Shared memory 206 can be configured to store data in response to input requests and / or present data in response to output requests (e.g., requests from processor 158, DRAM 150, scheduler circuit 190a, and / or hardware engines 204a-204n). In an example, shared memory circuit 206 can implement an on-chip memory for computer vision processor 102. Shared memory 206 is generally operable to store all or a portion of a multi-dimensional array (or vector) of input data elements and output data elements generated and / or utilized by hardware engines 204a-204n. Input data elements can be transferred from DRAM circuit 150 to shared memory 206 via memory bus 200. Output data elements can be sent from shared memory 206 to DRAM circuit 150 via memory bus 200.

[0128] Path 208 can implement a transfer path within processor 102. Transfer path 208 is generally operable to move data from scheduler circuit 190a to shared memory 206. Transfer path 208 can also be operable to move data from shared memory 206 to scheduler circuit 190a.

[0129] Processor 158 is shown in communication with computer vision processor 102. Processor 158 can be configured as a controller for computer vision processor 102. In some embodiments, processor 158 can be configured to transmit instructions to scheduler 190a. For example, processor 158 can provide one or more directed acyclic graphs to scheduler 190a via DAG memory 202. Scheduler 190a can initialize and / or configure hardware engines 204a-204n in response to parsing the directed acyclic graphs. In some embodiments, processor 158 can receive status information from scheduler 190a. For example, scheduler 190a can provide processor 158 with status information and / or preparation status of outputs from hardware engines 204a-204n to enable processor 158 to determine one or more next instructions to execute and / or decisions to make. In some embodiments, processor 158 can be configured to communicate with shared memory 206 (e.g., directly or through scheduler 190a, which receives data from shared memory 206 via path 208). Processor 158 can be configured to retrieve information from shared memory 206 to make decisions. Instructions executed by processor 158 in response to information from computer vision processor 102 can vary according to design criteria of a particular implementation.

[0130] Reference Figure 5FIG. 3, shows a diagram illustrating monocular structured light depth estimation. A scene 250 is shown. The scene 250 can include a camera device 100, an object 252, and a reference location 254. A structured light projector 106, a lens 160, a structured light pattern lens 162, and an image sensor 180 are shown. Other components of the camera device 100 are omitted for clarity.

[0131] A location DC is shown at the camera device 100. The location DC can represent a baseline location of the lens 160 and the structured light pattern lens 162. A location DO is shown. The location DO can represent a distance of the object 252 from the baseline location DC of the camera device 100. In an example, the object 252 can be DO from the camera device 100. A location DR is shown. The location DR can represent a distance of the reference location 254 from the baseline location DC of the camera device 100. In an example, the reference location 254 can be DR from the camera device 100.

[0132] The object 252 is shown at a distance DO from the baseline location DC. The object 252 can be located at a distance DO that can be closer to the lens 160 than the reference location 254. For example, the object 252 can be at some distance DO between the baseline location DC and the reference location DR. The object 252 is shown at some location between the structured light projector lens 162 and the lens 160. For example, the object 252 is shown offset from both the structured light projector lens 162 and the lens 160. The object 252 can represent a person, an animal, a vehicle, an inanimate object (e.g., a curb, a sign, a bicycle, a tree, a box, etc.), and the like. The type, size, shape, distance from the camera device 100, and / or distance from the reference location 254 of the object 252 can vary according to design criteria of a particular implementation.

[0133] The reference location 254 can be an example of content in a reference image. The reference location 254 can be at a distance from the camera device 100 that is within a range of the structured light SLP that can be generated by the structured light projector 106. The reference location 254 can be used by the processor 102 as a basis for comparison of incoming video frames. In an example, a newly incoming video frame can be compared to a reference video frame. The reference video frame can include images of the structured light pattern SLP at various distances (e.g., the reference location 254 can be one of the various distances of the reference image). The processor 102 can be configured to compare the newly incoming video frame to the reference image using computer vision operations and / or 3D reconstruction to determine a location where the object 252 is located. The reference image can enable performance of 3D reconstruction when monocular lens 160 is implemented. While one reference location 254 is shown in the scene 250, the processor 102 can be configured to capture multiple reference images of various reference locations at different distances to have a basis for comparing the structured light pattern SLP. The number of stored reference images and / or the various distances of the reference locations can vary according to design criteria of a particular implementation.

[0134] A line 260 is shown. The line 260 can represent a baseline depth of the camera device 100 from the object 252. The line 260 can illustrate a depth direction. The CNN module 190b of the processor 102 can be configured to arrange the reference images along the illustrated depth direction 260. A line 262 is shown. The line 262 can represent the structured light pattern SLP generated by the structured light projector 106. The structured light pattern SLP is shown as the line 262 projected onto the object 252. A line 264 is shown. The line 264 can represent the structured light pattern SLP captured by the image sensor 180 relative to the object 252 and the depth direction of the object 252. A line 266 is shown. The line 266 can represent the structured light pattern SLP and is generated by the structured light pattern of the reference image. The structured light pattern SLP is shown as the line 266 projected onto the reference location 254 (e.g., captured when the object 252 is not present). For example, instead of being projected onto the object 252, the structured light pattern SLP can project beyond the distance DO and onto the reference location 254 when the object 252 is not present. A line 268 is shown. The light 268 can represent the structured light pattern SLP captured by the image sensor 180 relative to the reference location 254 and the depth direction of the reference location 254.

[0135] When the object 252 is present, the structured light pattern SLP can be projected onto the object 252 (e.g., line 262) and the image sensor 180 can capture the object 252 with the structured light pattern SLP (e.g., line 264). The line 264 can illustrate that the structured light pattern SLP projected onto the object 252 can enter the lens 160 and be captured by the image sensor 180 at location A. When the object 252 is not present, the structured light pattern SLP can be projected onto the reference location 254 (e.g., line 266) and the image sensor 180 can capture the reference location 254 with the structured light pattern SLP (e.g., line 268). The line 268 can illustrate that the structured light pattern SLP projected onto the reference location 254 can enter the lens 160 and be captured by the image sensor 180 at location B.

[0136] The distance D is shown on the image sensor 180. The distance D can represent the distance between location A (e.g., where the structured light pattern SLP is captured for the object 252) and location B (e.g., where the structured light pattern SLP is captured for the reference location 254). The distance D can represent the disparity between the structured light pattern SLP for the distance DO of the object 252 and the distance DR of the reference location 254. The amount of disparity D can be used by the processor 102 and / or the CNN module 190b to determine a depth estimate for the object 252 using monocular structured light patterns.

[0137] In various depth imaging devices consistent with the scope of the present disclosure, the number and / or type of cameras can vary. Broadly, one or more cameras can be configured to provide video from which a time-resolved sequence of three-dimensional depth maps can be obtained via subsequent processing. As used herein, the term “depth map” refers to an array of pixels registered to respective regions of an imaged scene, each pixel’s depth value indicating a distance between the camera and a surface imaged by the particular pixel.

[0138] In other embodiments, stereo camera technology or “time-of-flight” (ToF) sensors can be implemented to determine a depth estimate for the object 252. In an example, a depth imaging device can be implemented that can include a left stereo camera and a right stereo camera. Time-resolved images from the two cameras can be registered to one another and combined to produce a depth-resolved video. In another example, a ToF depth camera can be implemented that includes a light source configured to project pulsed infrared illumination onto a scene. In an example, two cameras can be configured to detect pulsed illumination reflected from the scene. In an example, the cameras can include electronic shutters that are synchronized with the pulsed illumination, but the integration times of the cameras can differ, such that a pixel-resolved time-of-flight of the pulsed illumination, from the light source to the scene and then to the cameras, can be discerned from the relative amounts of light received in respective pixels of the two cameras.

[0139] In various applications, monocular structured light depth estimation, stereo vision depth estimation, or time-of-flight depth estimation can be used to generate a high resolution depth map image that is aligned with a high resolution 2D image using accelerated alignment of high resolution images and depth maps for low bit-width floating point representation according to embodiments of the present application.

[0140] Referring to Figure 6 , Figure 6 A diagram illustrating decomposition of high resolution 2D and depth map images into sub-images that can be used to compute alignment matrices using low bit-width floating point representation is shown. In an example, a high resolution 2D image I is illustrated by array 300, while a depth map image D is illustrated by array 302. By splitting each of arrays 300 and 302 into a plurality of component sub-arrays, high resolution 2D image I and depth map image D can each be decomposed into a plurality of sub-images. The sub-images can be used to compute alignment matrices using low bit-width floating point representation and operations.

[0141] To address the problem of depth alignment of high resolution images in low bit-width floating point operation platforms, a depth alignment algorithm based on a combination of matrix decomposition, offset, and table lookup can be implemented. A floating point unit (FPU) can increase the range and precision of mathematical computations, or achieve greater throughput in a shorter time. An FPU can be used to more easily meet real-time constraints. An FPU can enable a system to complete computations in a shorter time, and thus, spend more time in an idle (e.g., sleep) mode. By enabling a system to spend more time in an idle mode, a system with an FPU can save power and extend battery life. A floating point number is typically represented in the form of Ax Base B , where A is a mantissa and B is an exponent. Mantissa A typically represents the significant digits of a number. Depending on exponent B, the mantissa can represent an integer or a fraction. In an example, a single precision floating point representation includes 32 bits, which can be arranged as shown in Table 1 below, with the most significant bit (MSB) on the left:

[0142] Table 1

[0143] 31 30 23 22 0 1 11111111 11111111111111111111111 Symbol Exponent Mantissa

[0144] The sign of a binary floating point number is represented by a single bit. A 1 bit indicates a negative number, while a 0 bit indicates a positive number. A double precision floating point representation typically includes 64 bits, 1 bit for the sign, 11 bits for the exponent, and 52 bits for the mantissa.

[0145] In an example, for a depth alignment problem on a binary signed floating point operation platform with a sufficient number of precision bits and a k-bit mantissa, where k is greater than 2, the resolution of both depth image D and 2D image I can be u0 x v0. Generally, when 2 AThe number of precision bits A is considered sufficient when greater than or equal to the particular range of variable to be represented. In an example, for a range of variable values from 0.0 to 1280.0 to be represented by a floating point number, the number of precision bits (or mantissa) of the floating point number should be equal to or greater than 11 bits (e.g., 2 11 >1280). In another example, for a variable value 320.1 to be described sufficiently accurately by a floating point number, the number of precision bits (or mantissa) of the floating point number should be equal to or greater than 4 bits (e.g., 2 -4 <0.1). Generally, the more accurate the result, the greater the number of precision (or mantissa) bits required.

[0146] In an example, the camera intrinsic parameters corresponding to the high resolution depth image D can include focal length f Dx , f Dy and principal point offset c Dx , c Dy . In an example, the camera intrinsic parameters corresponding to the high resolution 2D image I can be focal length f Ix , f Iy and principal point offset c Ix and c Iy . In an example, the camera projection matrix of the camera can be T (T e R 4x4 ) and the depth range in the high resolution depth image D can be d e (d min , d max ), where d min is greater than 1. Assuming the depth camera and the 2D camera are close enough and the included angle is small enough, the implementation steps can be implemented as follows.

[0147] In the first step, the value l can be calculated as l = 2 k-1 . The value l can be used to equally divide the depth image D and the 2D image I into r rows and c columns, for a total of r x c pieces of component images (sub-images), where

[0148]

[0149] and where, denotes the ceiling function. For each r x c pieces of sub-images, the sub-image in the i-th row and j-th column (i = 0,..., r - 1, j = 0,..., c - 1) can be denoted as D ij and I ij respectively, and the respective width u ij and height v ij can be calculated according to the following equations respectively:

[0150]

[0151] In the next step, for each pair of sub-images Dij ,I ij The intrinsic matrix of the camera for each sub-image can be computed as follows:

[0152]

[0153] The intermediate matrix A ij (A ij ∈ R 4x4 ) can be computed as follows:

[0154]

[0155] where k b = min(8, d min ). The first three rows of the intermediate matrix A ij (A ij ∈ R 4x4 ) can then be recorded as the disparity-depth matrix Q ij (Q ij ∈ R ij ) for each pair of sub-image D ij , I 3x4 . In the third step, a new depth image can be obtained.

[0156] In the fourth step, the projection matrices p ij and p ij can be generated for the depth image sub-images D Dij and 2D sub-images I Iij respectively. For each depth image sub-image D ij , for each pixel (u Dij , v Dij ), and the corresponding depth value at the pixel, the tensors of the projection matrices p Dij and p Iij can be generated according to the following equations:

[0157]

[0158] p Iij = Q ij · p Dij = [x Iij y Iij z Iij ] T . Equation 9

[0159] In the fifth step, the depth value for each pixel of each 2D sub-image I ij can be computed using the following equation:

[0160]

[0161] Then, in the high resolution 2D image I, the depth value assigned to each pixel (u I , v I ) can be determined by lookup in the corresponding 2D sub-image, as follows:

[0162] (u I , v I ) = (u Iij + j x u ij , v Iij + i x v ij ) and

[0163] the depth value is If there are more depth images, the process can return to the third step. If there are no new depth images, the algorithm can terminate.

[0164] Referring to Figure 7 , Figure 7 a diagram of a data arrangement for computing alignment matrices using low-bit- width floating point representation is shown. A data arrangement 350 is shown. The processor 102 can be configured to execute the data arrangement 350 to implement accelerated alignment of high resolution 2D image and high resolution depth map images using low-bit-width floating point representation. The alignment techniques can be implemented to perform 3D reconstruction locally on the processor 102 and / or using a processor that implements low-bit-width floating point operations (e.g., perform all processing on the edge device 100 without uploading data to a cloud computing resource to perform analysis).

[0165] The data arrangement 350 can be executed by the processor 102 and / or the CNN module 190b. The data arrangement 350 can be executed offline (e.g., during offline alignment) or in real-time. In an example, the images can be acquired and pre-processed prior to pixel data of the input video images being generated by the capture device 104 and arranged by the processor 102 into video frames. The processor 102 and / or the CNN module 190b can be configured to arrange image data obtained after image acquisition and pre-processing.

[0166] In the data arrangement 350, 2D sub-images 352a-352n and depth map sub-images 354a-354n are shown. To perform the data arrangement 350, the processor 102 and / or the CNN module 190b can be configured to perform matrix decomposition offsets to ensure that each of the decomposed images 352a-352n and 354a-354n can be represented by a low-bit-width representation without loss of precision. The specialized hardware modules 204a-204n can be configured to decompose the high resolution 2D image I to generate the sub-images 352a-352n and the high resolution depth map image D to generate the sub-images 354a-354n. For example, the decomposition can be performed according to the number of mantissa bits of the low-bit-width floating point representation of the processor 102.

[0167] In data arrangement 350, intrinsic matrices of cameras 356a-356n and 358a-358n are shown for each 2D sub-image and each depth sub-image, respectively. To perform data arrangement 350, processor 102 and / or CNN module 190b can be configured to perform matrix decomposition offsets to ensure that each of decomposed images 352a-352n and 354a-354n can be used in low-bitwidth operations without loss of precision. Processor 102 and / or CNN module 190b can be configured to use dedicated hardware modules 204a-204n to calculate intrinsic matrices of cameras 356a-356n and 358a-358n according to Equations 5 and 6 above.

[0168] In data arrangement 350, projection matrices of camera 360 and Q matrices 362a-362n are also shown. Processor 102 and / or CNN module 190b can be configured to use dedicated hardware modules 204a-204n to calculate Q matrices 362a-362n using intrinsic matrices of cameras 356a-356n and 358a-358n, projection matrices of camera 360, and Equation 7 above. When data arrangement 350 has been generated, processor 102 and / or CNN module 190b can perform accelerated alignment of high-resolution 2D images and high-resolution depth map images using low-bitwidth floating-point representations. In an example, processor 102 and / or CNN module 190b can perform accelerated alignment of high-resolution 2D images and high-resolution depth map images by applying Equations 8, 9, and 10 above to obtain depth values for each pixel of the high-resolution 2D images.

[0169] Referring to Figure 8 , Figure 8 A flowchart illustrating a method (or procedure) 400 is shown. Method 400 can be implemented to accelerate alignment of high-resolution 2D images and high-resolution depth images using low-bitwidth floating-point representations. Floating-point representations allow computations to be performed using floating-point units (FPUs). Floating-point units can increase the range and precision of mathematical computations and / or increase throughput in a shorter amount of time. Increased throughput and reduced time generally make it easier to meet real-time constraints. Moreover, by enabling a system to complete computations in a shorter amount of time, power costs can be reduced, enabling low-power operation and extending battery life. Method 400 generally includes step (or state) 402, step (or state) 404, step (or state) 406, step (or state) 408, decision step (or state) 410, step (or state) 412, step (or state) 414, and step (or state) 416.

[0170] Method 400 generally begins at step 402 and moves to step 404. In step 404, method 400 can determine the number of significand bits of the floating point format of the camera system. In step 406, method 400 can determine the number of rows (e.g., r) and columns (e.g., c) for splitting the high resolution 2D image and the high resolution depth map image into a respective plurality of decomposed (or sub) images, which can be arranged as an r x c array of sub-images. In step 408, method 400 can compute a respective camera intrinsic parameter matrix and a respective Q matrix for each sub-image location. In step 410, method 400 can input the high resolution depth map image and move to step 412.

[0171] In step 412, method 400 can compute the projection coordinates in the respective high resolution 2D image and the depth for each depth map sub-image D ij and for each depth. Method 400 can then transform the high resolution depth map image to align with the original high resolution 2D image. In step 414, method 400 can determine whether a new depth map image is available. When there is a new depth map image, method 400 can move to step 410. When there is no new depth map image, method 400 can move to step 416 and terminate.

[0172] In various embodiments, the alignment matrix for a number of uniform small images can be solved by matrix decomposition and offset, such that the alignment accuracy of the high resolution 2D image and the high resolution depth map image using low bit-width floating point representation can be maintained. After each small resolution image is obtained by decomposing the large resolution image, a corresponding coordinate transformation can be performed on each small resolution image. In an example using 1280 x 960 RGB and depth map images, the alignment process according to embodiments of the present disclosure can obtain 16 equal parts (or sub-images) of 320 x 240 pixels, a 3 x 3 intrinsic parameter matrix of the 2D image generation device, a 3 x 3 intrinsic parameter matrix of the depth map generation device, an offset of the 2D RGB image relative to the depth map generation device, and a 4 x 4 rotation matrix for implementing matrix decomposition offset to accommodate small resolution for improved accuracy. Finally, an aligned depth map image can be generated.

[0173] In an example, a point cloud can be generated from an original depth map image captured by a depth sensor. The point cloud can be re-projected to a corresponding RGB image. For each position (u, v) and depth of each point in the point cloud, a corresponding position (u, v) and depth in the RGB image can be obtained. An aligned depth map image can be generated by collecting the position (u, v) and depth of each point in the point cloud in the aligned depth map image.

[0174] In an example, an array for storing a blank depth image can be created. For each point in the point cloud, the depth of each point can be assigned to a pixel (u, v) in the blank depth image array. In an example, the depth of each point in the blank depth image array can be determined by looking up the depth of the corresponding point in the corresponding aligned depth sub-image. When the depth of all points of the point cloud are assigned to the pixels of the blank depth image, the blank depth image becomes an aligned depth image.

[0175] Reference Figure 9 , Figure 9 An image 500 illustrating an example of an alignment result obtained by double-precision floating point operation is shown. For a depth image of 1280x960, the alignment result obtained by double-precision floating point operation can be taken as a reference value. The precision and speed of data using double-precision floating point operation at a resolution of 1280x960 can be considered as a true value result of the depth alignment operation.

[0176] Reference Figure 10 , Figure 10 An image 600 illustrating an example of an alignment result obtained by single-precision floating point operation is shown. For a depth image of 1280x960, the alignment result obtained by single-precision floating point operation can be shown as in the image 600. The precision and speed of data alignment using single-precision floating point operation at a resolution of 1280x960 can be summarized in the following Table 1:

[0177] Table 1

[0178] Dimension Average_Error Max_Error Min_Error u I ]]> 6.1e-05 pixels 1 pixel 0 pixels v I ]]> 1.8e-05 pixels 1 pixel 0 pixels d I ]]> 8.0e-05 mm 1 mm 0 mm

[0179] The time cost is 100 milliseconds / frame.

[0180] Reference Figure 11 , Figure 11 An image 700 is shown, which shows another example of an alignment result using image splitting technique according to an embodiment of the present application. When the accelerated alignment of high-resolution 2D image and high-resolution depth image is implemented based on the combination of matrix decomposition offset and table lookup without low-bit-width floating point representation, the precision and speed of data operation using half-precision floating point operation at a resolution of 1280x960 can be summarized in the following Table 2:

[0181] Table 2

[0182]

[0183]

[0184] The time cost is 16 milliseconds / frame.

[0185] It can be observed in the results of the depth alignment without the matrix decomposition offset and table lookup combination technique shown in image 700 that the results include gaps (e.g., shown by boxes 702 and 704) and the texture is blurred (e.g., shown by box 706).

[0186] Referring to Figure 12 , Figure 12 An image 800 is shown that illustrates an example of aligned results using low- bit-width floating-point representations based on the matrix decomposition offset and lookup table combination according to embodiments of the present application. When implementing accelerated alignment of high-resolution 2D image and high-resolution depth map images using low-bit-width floating-point representations based on the matrix decomposition offset and lookup table combination, the precision and speed of data operations using half-precision floating-point operations with a resolution of 1280x960 can be summarized in Table 3 below:

[0187] Table 3

[0188] Dimension Average_Error Max_Error Min_Error u I ]]> 0.0345 pixels 1 pixel 0 pixels v I ]]> 0.0289 pixels 1 pixel 0 pixels d I ]]> 0.380 mm 2 mm 0 mm

[0189] The time cost is 26 milliseconds / frame.

[0190] It can be observed in image 800 that the results of the depth alignment using the matrix decomposition offset and table lookup combination are improved (e.g., shown by boxes 802, 804, and 806). From the above error statistics, it can be seen that the method according to embodiments of the present application can effectively reduce errors when using half-precision floating-point operations compared to methods that do not use low-bit-width floating-point representations based on the matrix decomposition offset and lookup table combination. It can also be seen that reduced errors are obtained relative to single-precision and double-precision operations while still providing increased speed.

[0191] The functions performed by the Figures 1-8 The functions performed by the

[0192] The application can also be implemented by preparing an ASIC (application specific integrated circuit), a platform ASIC, an FPGA (field programmable gate array), a PLD (programmable logic device), a CPLD (complex programmable logic device), a gate array, an RFIC (radio frequency integrated circuit), an ASSP (application specific standard product), one or more monolithic integrated circuits, one or more chips or dies arranged as a flip chip module and / or a multi-chip module, or by interconnecting one or more appropriate conventional circuit networks as described herein, the modifications of which will be apparent to those skilled in the art.

[0193] The application can therefore also include a computer product, which can be a storage medium or media and / or transmission medium or media, comprising instructions that can be used to program a machine to perform one or more processes or methods in accordance with the application. The operation of the machine executing the instructions contained in the computer product and the surrounding circuitry, can transform input data into one or more files that are stored on storage media and / or one or more output signals representing a physical object or substance, such as audio and / or visual descriptions. The storage medium can include, but is not limited to, any type of disk including floppy disks, hard disks, magnetic disks, optical disks, CD-ROMs, DVDs, and magneto-optical disks, and circuitry such as ROM (read only memory), RAM (random access memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), UVPROM (ultraviolet erasable programmable ROM), flash memory, magnetic or optical cards, and any type of media suitable for storing electronic instructions.

[0194] The elements of the application can form part or all of one or more devices, units, components, systems, machines and / or apparatuses. These devices can include, but are not limited to, servers, workstations, storage array controllers, storage systems, personal computers, laptop computers, notebook computers, palm computers, cloud servers, personal digital assistants, portable electronic devices, battery-powered devices, set-top boxes, encoders, decoders, transcoders, compressors, decompressors, pre-processors, post-processors, transmitters, receivers, transceivers, cryptographic circuits, cellular telephones, digital cameras, positioning and / or navigation systems, medical devices, head-up displays, wireless devices, audio recording, audio storage and / or audio playback devices, video recording, video storage and / or video playback devices, gaming platforms, peripheral devices and / or multi-chip modules. Those skilled in the relevant art will appreciate that the elements of the application can be implemented in other types of devices to meet the standards of a particular application.

[0195] The terms “may” and “generally,” when used herein in conjunction with the verb “to be” are intended to convey the following meaning: the description is exemplary and is considered broad enough to encompass both the specific examples presented in this disclosure and alternative examples that can be derived from this disclosure. The terms “may” and “generally” used herein should not be interpreted as necessarily implying the desirability or possibility of omitting the corresponding element.

[0196] The designation of various components, modules, and / or circuits “a” - “n,” when used herein, discloses a single component, module, and / or circuit or a plurality of such components, modules, and / or circuits, where the designation “n” is used to mean any particular integer. Each different component, module, and / or circuit having the designation of “a” - “n” can indicate that different components, modules, and / or circuits can have a matching number of instances or a different number of instances. The instance designated “a” can represent the first of a plurality of instances, and the instance “n” can refer to the last of a plurality of instances, without implying a particular number of instances.

[0197] While the application has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the scope of the application.

Claims

1. A computing device comprising: an interface configured to receive pixel data; and a processor configured to: (i) process the pixel data arranged as video frames, (ii) decompose each of the video frames comprising a high resolution two-dimensional (2D) image into a plurality of 2D sub-images using a decomposition offset, (iii) decompose each of the video frames comprising a high resolution depth map image into a plurality of depth map sub-images using the decomposition offset, (iv) compute a respective alignment matrix for each of the plurality of 2D sub-images and a respective one of the plurality of depth map sub-images using the decomposition offset, (v) perform matrix operations on the plurality of 2D sub-images and the plurality of depth map sub-images using the respective alignment matrices to obtain a plurality of aligned 2D sub-images and depth map sub-images, (vi) determine projection coordinates and depth values for the plurality of aligned 2D sub-images and depth map sub-images based on the high resolution 2D image, and (vii) generate an aligned high resolution depth map using the projection coordinates and the depth values, wherein: (a) the video frames have pixel coordinate values that are greater than a number of bits supported by a floating point operator of the processor, and (b) sizes of the plurality of 2D sub-images and depth map sub-images are determined based on the floating point operator of the processor.

2. The computing device of claim 1, wherein, the decomposition offset is configured to ensure that each of the sub-images is represented as a low-bit floating point representation without loss of precision.

3. The computing device of claim 2, wherein, the processor is configured to compute the decomposition offset based on a number of significand bits of the low-bit floating point representation.

4. The computing device of claim 1, wherein, the high resolution two-dimensional (2D) image and the high resolution depth map image are split into a plurality of rows and a plurality of columns of sub-images based on the decomposition offset.

5. The computing device of claim 1, wherein, the processor is further configured to compute a respective camera matrix for each of the 2D sub-images and each of the depth map sub-images.

6. The computing device of claim 5, wherein, the processor is further configured to compute a respective intermediate matrix for each of the respective 2D sub-images and depth map sub-images using the respective camera matrix of each respective 2D sub-image, the respective camera matrix of each respective depth map sub-image, and a camera projection matrix of the computing device.

7. The computing device of claim 6, wherein, the processor is further configured to compute a respective disparity-depth matrix for each of the respective 2D sub-images and depth map sub-images using the intermediate matrix.

8. The computing device of claim 7, wherein, the processor is further configured to generate an aligned high resolution depth map for the high resolution 2D image using the respective disparity-depth matrix of the respective 2D sub-images and depth map sub-images.

9. The computing device of claim 1, wherein, the processor comprises a plurality of hardware engines configured to perform the matrix operations.

10. The computing device of claim 1, wherein, the processor comprises a system on a chip.

11. A method for accelerated alignment of 2D images and depth images using low-bit width floating point representations, the method comprising: receiving pixel data; processing the pixel data arranged as video frames; each of the video frames comprising high resolution two-dimensional (2D) images is decomposed into a plurality of 2D sub-images using a decomposition offset; each of the video frames comprising high resolution depth map images is decomposed into a plurality of depth map sub-images using the decomposition offset; a respective alignment matrix is computed for each of the plurality of 2D sub-images and a respective one of the plurality of depth map sub-images using the decomposition offset; a matrix operation is performed on the plurality of 2D sub-images and the plurality of depth map sub-images using the respective alignment matrices to obtain a plurality of aligned 2D sub-images and depth map sub-images; projection coordinates and depth values of the plurality of aligned 2D sub-images and depth map sub-images are determined based on the high resolution 2D images; and an aligned high resolution depth map is generated using the projection coordinates and depth values, wherein: (a) the video frames have pixel coordinate values that are larger than a number of bits supported by a floating point operator of a processor, and (b) sizes of the plurality of 2D sub-images and depth map sub-images are determined based on the floating point operator of the processor.

12. The method of claim 11, wherein, the decomposition offset is configured to ensure that each of the sub-images is represented as a low-bit floating point representation without loss of precision.

13. The method of claim 12, further comprising: the decomposition offset is computed based on a number of bits of a mantissa of the low-bit floating point representation.

14. The method of claim 11, wherein, the high resolution two-dimensional (2D) images and the high resolution depth map images are split into a plurality of rows and a plurality of columns of sub-images based on the decomposition offset.

15. The method of claim 11, further comprising: a respective camera matrix is computed for each of the 2D sub-images and each of the depth map sub-images.

16. The method of claim 15, further comprising: a respective intermediate matrix is computed for each of the respective 2D sub-images and depth map sub-images using the respective camera matrix of each respective 2D sub-image, the respective camera matrix of each respective depth map sub-image, and a camera projection matrix of a device.

17. The method of claim 16, further comprising: a respective disparity-depth matrix is computed for each of the respective 2D sub-images and depth map sub-images using the intermediate matrix.

18. The method of claim 17, further comprising: an aligned high resolution depth map is generated for the high resolution 2D images using the respective disparity-depth matrix of the respective 2D sub-images and depth map sub-images.

19. The method of claim 11, further comprising: the matrix operation is performed using a plurality of hardware engines of a processor circuit.

20. The method of claim 19, wherein, the processor circuit is part of an edge device.

Images