Disparity map construction using guide nodes

Abstract

A device including an interface and a processor. The interface may be configured to receive pixel data. The processor may be configured to: generate a reference image and a target image according to the pixel data; perform a disparity operation on the reference image and the target image; and construct a disparity map in response to the disparity operation. The disparity operation may include: selecting a guide node from the pixel data, the guide node including a pixel and a plurality of surrounding pixels; determining a peak position for the pixel by performing a full range search; calculating a shift offset peak position for each of the surrounding pixels by performing a block matching operation in a local range near the peak position; and generating a value in a disparity map for the pixel data in response to the peak position for the pixel and the shift offset peak position for each of the surrounding pixels.

Description

Technical Field

[0001] The present invention relates generally to stereoscopic vision, and more particularly to methods and / or apparatus for implementing disparity map construction using guidance nodes. Background Art

[0002] Stereo vision is a widely known technology. Stereo vision utilizes a pair of cameras located at known distances and angles relative to each other. By comparing the disparity between two images generated by the camera pair using the known distance and angle between the cameras, valuable information such as the distance to an object can be determined. Stereo vision is widely used in technologies such as 3D reconstruction, autonomous driving, stereo object recognition, etc.

[0003] Disparity maps are a widely popular option for displaying stereo vision. A disparity map is a 2D vector that shows the difference in horizontal axis positioning between one pixel in a reference image and the matching pixel in the target image. For example, the target image and the reference image are offset from each other based on the known distance and angle between the camera pair, resulting in similar pixel data being located at different locations. A pixel at location x0, y in the reference image may have a disparity of x1-x0 with the same pixel at location x1, y in the target image.

[0004] Generating a disparity map is computationally expensive, which limits the real-time availability of the disparity map and / or has high power requirements for the implementation. In particular, for real-time applications such as autonomous driving, the disparity map should be generated quickly and accurately. Conventional techniques for generating disparity maps have a trade-off between rapid matching and accurate matching. A conventional method for constructing a disparity map is block matching, which is simple and fast, but produces a large number of mismatches (i.e., noisy disparity maps). Another method for constructing a disparity map is semi-global block matching, which has fewer mismatches, but is complex and slow to operate (e.g., limited use). Block matching and semi-global block matching can only construct a disparity map with pixel-level accuracy, rather than constructing a disparity map with sub-pixel accuracy without post-processing. Constructing a disparity map with pixel-level accuracy may cause waveform distortion in the disparity map.

[0005] It would be desirable to implement disparity map construction using guide nodes. Summary of the invention

[0006] The present invention relates to a device including an interface and a processor. The interface can be configured to receive pixel data. The processor can be configured to: generate a reference image and a target image according to the pixel data; perform a disparity operation on the reference image and the target image; and construct a disparity map in response to the disparity operation. The disparity operation may include: selecting a guide node from the pixel data, the guide node including a guide node pixel and a plurality of surrounding pixels; determining a guide node pixel peak position for the guide node pixel by performing a full range search in the reference image and the target image; calculating a shift offset peak position for each of the surrounding pixels in the guide node by performing a block matching operation in a local range near the guide node pixel peak position; and generating a value in a disparity map for the pixel data in the guide node in response to the guide node pixel peak position for the guide node pixel and the shift offset peak position for each of the surrounding pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present invention will become apparent from the following detailed description, appended claims and accompanying drawings.

[0008] Figure 1 is a diagram showing a stereo camera in a vertical orientation.

[0009] Figure 2 is a diagram showing a stereo camera in a horizontal orientation.

[0010] Figure 3 is a block diagram illustrating an example embodiment of the present invention.

[0011] Figure 4 is a diagram showing the disparity between a reference image and a target image.

[0012] Figure 5 is a diagram illustrating the generation of disparity values ​​using a guide node.

[0013] Figure 6 is a diagram showing pixels in a guide node.

[0014] Figure 7 is a diagram showing a full search range and a local search range.

[0015] Figure 8 is a diagram showing guiding nodes on a reference video frame.

[0016] Fig. 9 is a diagram showing a disparity map.

[0017] Fig.10 A diagram showing a stereoscopic image.

[0018] Fig.11 is a flow chart illustrating a method for constructing a disparity map using guide nodes.

[0019] Fig.12 is a flow chart illustrating a method for calculating disparity values ​​for surrounding pixels.

[0020] Fig.13 is a flow chart illustrating a method for calculating a disparity value for a center pixel. DETAILED DESCRIPTION

[0021] Embodiments of the present invention include providing a disparity map construction using guide nodes, which can (i) quickly construct a disparity map; (ii) accurately construct a disparity map; (iii) construct a disparity map at a sub-pixel level; (iv) eliminate waveform distortion in stereo vision; (v) achieve real-time stereo vision with low power consumption; (vi) use guide nodes to achieve local range matching for surrounding pixels; (vii) model a parabola used to calculate a shift offset between a reference image and a target image; (viii) be 20-30 times faster than a block matching operation by reducing computational complexity; (ix) perform a full range search on a center pixel of a guide node and perform a local range search on surrounding pixels near a peak position of the guide node; and / or (x) be implemented as one or more integrated circuits.

[0022] Embodiments of the present invention may be configured to operate in a system implementing stereoscopic vision. A disparity map may be generated by using a guide node. The guide node may limit the amount of calculations performed to determine a shift offset between a reference image and a target image. The reference image and the target image may be video frames generated by a stereo camera (e.g., a camera comprising a pair of lenses arranged in a predetermined orientation).

[0023] Embodiments of the present invention may be configured to construct a disparity map with sub-pixel accuracy. Constructing a disparity map with sub-pixel accuracy may prevent waveform distortion. A disparity map generated using a guide node may achieve sub-pixel accuracy without post-processing.

[0024] Compared to performing a full range search to determine the shift offset between a reference image and a target image, using a guide node to construct a disparity map can reduce the amount of calculations performed. Compared to block matching (e.g., BM) techniques (e.g., using a full range search for each pixel), using a guide node can enable a disparity map to be constructed 20 to 30 times faster (e.g., constructed in less time). Compared to BM techniques, using a guide node can also generate less noise in the disparity map. The guide node can reduce and / or eliminate waveform distortion in stereoscopic vision performed in response to the disparity map.

[0025] Embodiments of the present invention may implement a guide node to determine a peak position for a local range match of surrounding pixels. Local range matching may be implemented to reduce the number of calculations compared to a full range search while preventing mismatches when determining a shift offset between a reference image and a target image. Preventing mismatches (e.g., reducing the number of mismatches compared to a BM technique) may enable a disparity map to be constructed with less noise compared to a BM technique.

[0026] Local range matching can be achieved in response to determining the peak position of the guide node. A full range search can be performed to determine the peak position of the center pixel of the guide node (e.g., the guide node peak position). A local range search can be performed for each of the pixels around the center pixel of the guide node (e.g., the surrounding pixels). The local range search can be performed on multiple pixel blocks. In an example, three pixel blocks can be used for the local range search. The pixel blocks can be compared between the target image and the reference image to determine the matching score. The matching score can be used to model a parabola. The peak of the modeled parabola can be used to calculate the matching position. The matching position can be used to determine the disparity value used to construct the disparity map. The calculated disparity value can be sub-pixel level.

[0027] Performing a full range search for a center pixel in a guide node may include the same number of calculations as performing a full range search using a BM technique. Determining a peak position using a local range near a guide node peak position may include a smaller number of calculations than performing a full range search for each pixel in a reference image (e.g., using a BM technique).

[0028] refer to Figure 1 , a diagram illustrating a stereo camera in a vertical orientation is shown. An apparatus (or device or module) 100 is shown. The apparatus 100 can implement a camera system. The camera system 100 can implement a stereo camera. The stereo camera 100 can have different styles and / or use cases. For example, the stereo camera 100 can be an action camera, a ceiling-mounted security camera, a webcam, a hunting camera (trail camera), a car camera, a security camera, a battery-powered camera, a doorbell camera, etc. The design / style of the stereo camera 100 can vary according to the design criteria of a specific implementation.

[0029] The stereo camera 100 may include a block (or circuit) 102, blocks (or circuits) 104a-104b, and / or blocks (or circuits) 106a-106b. The circuit 102 may implement a processor. The circuits 104a-104b may implement a capture device. The circuits 106a-106b may implement a lens. The stereo camera 100 may include other components (not shown). Details of the components of the stereo camera 100 may be similar to those described in the accompanying drawings. Figure 3 A detailed description is given in association therewith.

[0030] The processor 102 may be configured to generate a video frame in response to the pixel data. The pixel data may be captured by the capture device 104a-104b. The processor 102 may be configured to perform various types of operations on the generated video frame. The processor 102 may be configured to detect objects and / or analyze the characteristics of the objects detected in the video frame. In an example, the processor 102 may be configured to perform computer vision operations to detect objects in the video frame (e.g., vehicles, trees, pedestrians, street signs, pets, debris on the lenses 106a-106b, etc.).

[0031] The capture devices 104a-104b and lenses 106a-106b are shown as being implemented as a stereo pair. The lenses 106a-106b may be implemented at a predetermined distance apart from each other. The lenses 106a-106b may be implemented as an internal angle (e.g., slightly toward the center of the stereo camera 100 so that straight lines projected from the lenses 106a-106b will eventually intersect at a predetermined distance away from the lenses 106a-106b). The predetermined distances and predetermined angles of separation of the lenses 106a-106b may be used by the processor 102 to perform parallax operations and / or calculations. Parallax operations may be used to generate a parallax map and / or a stereoscopic visual image. In one example, parallax operations may be used by the processor 102 to determine the distance of an object away from the stereo camera 100.

[0032] In the illustrated example, the lenses 106a-106b may be oriented vertically relative to each other (e.g., the lenses 106a and 106b may be vertically "stacked"). For example, the video frame generated in response to the pixel data captured by the top capture device 104a may be a top image of a stereo pair of images (e.g., a target video frame and a reference video frame), and the video frame generated in response to the pixel data captured by the bottom capture device 104b may be a bottom image of the stereo pair of images. The disparity map generated by the processor 102 using the lenses 106a-106b in the vertical orientation may be a vertical disparity map (e.g., a vertical disparity space image).

[0033] refer to Figure 2 , a diagram illustrating a stereo camera in a horizontal orientation is shown. An alternative embodiment of a stereo camera 100 is shown. The stereo camera 100 may include a processor 102, capture devices 104a-104b, and lenses 106a-106b. The stereo camera 100 is shown in a horizontal orientation.

[0034] In the example shown, the lenses 106a-106b can be oriented horizontally relative to each other (e.g., lens 106a can be a right lens and lens 106b can be a left lens). For example, the video frame generated in response to the pixel data captured by the right capture device 104a can be a right image of a stereo pair of images (e.g., a target video frame and a reference video frame), and the video frame generated in response to the pixel data captured by the left capture device 104b can be a left image of the stereo pair of images. The disparity map generated by the processor 102 using the lenses 106a-106b in the horizontal orientation can be a horizontal disparity map (e.g., a horizontal disparity space image).

[0035] The orientation of the stereo camera 100 may be selected based on the particular use case. Figure 1 The vertical orientation of the stereo camera 100 shown in association with Figure 2 The stereo camera 100 may be advantageously oriented in a horizontal manner compared to the horizontal orientation of the stereo camera 100 shown in the related drawings. Other orientations of the stereo camera 100 (e.g., diagonal orientations) may be implemented. In the illustrated example, the stereo camera 100 may include a stereo pair of cameras. In some embodiments, the stereo camera 100 may include multiple stereo pairs of cameras (e.g., a camera system of stereo pairs configured to simultaneously capture images in multiple fields of view). The orientation of the lenses 106a-106b may vary according to the design criteria of a particular implementation.

[0036] refer to Figure 3 , a block diagram illustrating an example embodiment of the present invention is shown. A block diagram of a camera system 100 is shown. The camera system 100 may be a camera system 100. Figure 1-2Representative examples of camera systems 100 are shown in association. Camera system 100 generally includes processor 102, capture devices 104a-104n, lenses 106a-106n, blocks (or circuits) 152, blocks (or circuits) 154a-154n, blocks (or circuits) 156, blocks (or circuits) 158a-158n, blocks (or circuits) 160, and / or blocks (or circuits) 162. Circuit 152 may implement a sensor. Circuit 154a-154n may implement a microphone (e.g., an audio capture device). Circuit 156 may implement a communication device. Circuit 158a-158n may implement an audio output device (e.g., a speaker). Circuit 160 may implement a memory. Circuit 162 may implement a power supply (e.g., a battery). Camera system 100 may include other components (not shown). In the example shown, some of components 106-158 are shown as being external to camera system 100. However, components 106-158 may be implemented within and / or attached to camera system 100 (e.g., better functionality may be provided if speakers 158a-158n are not located within the housing of camera system 100). The number, type, and / or arrangement of components of camera system 100 may vary depending on the design criteria of a particular implementation.

[0037] In an example implementation, the processor 102 may be implemented as a video processor. The processor 102 may include inputs 170a-170n and / or other inputs. The processor 102 may include an input / output 172. The processor 102 may include an input 174 and an input 176. The processor 102 may include an output 178. The processor 102 may include an output 180a and an input 180b. The number of inputs, outputs, and / or bidirectional ports implemented by the processor 102 may vary according to the design criteria of a particular implementation.

[0038] In the illustrated embodiment, the capture devices 104a-104n may be components of the camera system 100. In some embodiments, the capture devices 104a-104n may be separate devices configured to send data to the camera system 100 (e.g., separate devices remotely connected to the camera system 100, such as a drone, a robot, and / or a system of security cameras configured to capture video data). In one example, the capture devices 104a-104n may be implemented as part of an autonomous robot configured to patrol a particular path, such as a corridor. Similarly, in the illustrated example, the sensor 152, the microphones 154a-154n, the wireless communication device 156, and / or the speakers 158a-158n are shown as being external to the camera system 100, but in some embodiments, these components may be components of the camera system 100 (e.g., components within the camera system 100).

[0039] The camera system 100 may receive one or more signals (e.g., IMF_A-IMF_N), a signal (e.g., SEN), a signal (e.g., FEAT_SET), and / or one or more signals (e.g., DIR_AUD). The camera system 100 may present a signal (e.g., ENC_VIDEO) and / or a signal (e.g., DIR_AOUT). The capture devices 104a-104n may receive the signals IMF_A-IMF_N from the corresponding lenses 106a-106n. The processor 102 may receive the signal SEN from the sensor 152. The processor 102 may receive the signal DIR_AUD from the microphones 154a-154n. The processor 102 may present the signal ENC_VIDEO to the communication device 156 and receive the signal FEAT_SET from the communication device 156. For example, the wireless communication device 156 may be a radio frequency (RF) transmitter. In another example, the wireless communication device 156 may be a Wi-Fi module. In another example, the communication device 156 may be a device capable of RF transmission, Wi-Fi, Bluetooth, and / or other wireless communication protocols. In some embodiments, the signal ENC_VIDEO may be presented to a display device connected to the camera 100. The processor 102 may present the signal DIR_AOUT to the speakers 158a-158n.

[0040] The lenses 106a-106n may capture signals (e.g., IM_A-IM_N). The signals IM_A-IM_N may be images (e.g., simulated images) of the environment near the camera system 100 that are presented by the lenses 106a-106n as signals IMF_A-IMF_N to the capture devices 104a-104a. The lenses 106a-106n may be implemented as optical lenses. The lenses 106a-106n may provide zoom features and / or focus features. In one example, the capture devices 104a-104n and / or the lenses 106a-106n may be implemented as a single lens assembly. In another example, the lenses 106a-106n may be implemented as separate implementations from the capture devices 104a-104n. The capture devices 104a-104n are shown as being within the circuit 100. In an example implementation, the capture devices 104a - 104n may be implemented external to the circuit 100 (eg, as part of a lens / capture device assembly along with the lenses 106a - 106n ).

[0041] In some embodiments, two or more of the lenses 106a-106n may be configured as a stereo pair of lenses. For example, the camera 100 may implement stereoscopic vision. The lenses 106a-106n implemented as a stereo pair may be implemented at a predetermined distance apart from each other and at a predetermined interior angle. The predetermined distance and / or the predetermined interior angle may be used by the processor 102 to construct a disparity map for stereoscopic vision.

[0042] For a stereo camera implementation of the camera system 100, two capture devices (e.g., capture devices 104a-104b) having two corresponding lenses (e.g., lenses 106a-106b) can be implemented as a stereo pair. In some embodiments, the camera system 100 can include one stereo pair of lenses (e.g., lens 106a and lens 106b implemented as a stereo pair) and other lenses (e.g., lenses 106c-106n) that operate independently of each other. In some embodiments, each of the lenses 106a-106n can be paired with another of the lenses 106a-106n to implement multiple stereo pairs of cameras (e.g., to generate a panoramic video including multiple stereoscopic visual images). The number of capture devices 104a-104a and / or the number of lenses 106a-106n can vary according to the design criteria of a particular implementation.

[0043] The capture devices 104a-104n may be configured to capture image data for a video (e.g., signals IMF_A-IMF_N from the lenses 106a-106n). In some embodiments, the capture devices 104a-104n may be video capture devices such as cameras. The capture devices 104a-104n may capture data received through the lenses 106a-106n to generate raw pixel data. In some embodiments, the capture devices 104a-104n may capture data received through the lenses 106a-106n to generate a bitstream (e.g., to generate a video frame). For example, the capture devices 104a-104n may receive focused light from the lenses 106a-106n. The lenses 106a-106n may be oriented, tilted, panned, zoomed, and / or rotated to provide a target view from the camera system 100 (e.g., a view for a video frame, a view for a panoramic video frame captured using multiple capture devices 104a-104n, a target image and a reference image view for stereoscopic vision, etc.). Capture devices 104a-104n may generate signals (e.g., PIXELD_A-PIXELD_N). Signals PIXELD_A-PIXELD_N may be pixel data (e.g., a sequence of pixels that may be used to generate a video frame). In some embodiments, signals PIXELD_A-PIXELD_N may be video data (e.g., a sequence of video frames). Signals PIXELD_A-PIXELD_N may be presented to inputs 170a-170n of processor 102.

[0044] The capture devices 104a-104n may convert the received focused light signals IMF_A-IMF_N into digital data (e.g., a bit stream). In some embodiments, the capture devices 104a-104n may perform analog-to-digital conversion. For example, the capture devices 104a-104n may perform photoelectric conversion of the focused light received by the lenses 106a-106n. The capture devices 104a-104n may convert the bit stream into pixel data, images, and / or video frames. In some embodiments, the pixel data generated by the capture devices 104a-104n may be uncompressed and / or raw data generated in response to the focused light from the lenses 106a-106n. In some embodiments, the output of the capture devices 104a-104n may be a digital video signal.

[0045] Sensor 152 may include one or more input devices. Sensor 152 may be configured to detect physical input from the environment and convert the physical input into a computer-readable signal. Signal SEN may include a computer-readable signal generated by sensor 152. In an example, one of sensors 152 may be configured to detect the amount of light and present a computer-readable signal representing the amount of light detected. In another example, one of sensors 152 may be configured to detect motion and present a computer-readable signal representing the amount of motion detected. Sensor 152 may be configured to detect temperature (e.g., a thermometer), orientation (e.g., a gyroscope), movement speed (e.g., an accelerometer), etc. The type of input detected by sensor 152 may vary according to the design criteria of a specific implementation.

[0046] The data provided in the signal SEN provided by the sensor 152 can be read and / or interpreted by the processor 102. The processor 102 can use the data provided by the signal SEN to perform various operations. In some embodiments, the processor 102 can use the light read from the sensor 152 to determine whether to activate infrared light (e.g., to provide night vision). In another example, the processor 102 can use information about movement from an accelerometer and / or a gyroscope to perform motion correction on the generated video frames. The type of operation performed by the processor 102 in response to the signal SEN can vary according to the design criteria of the specific implementation.

[0047] The communication device 156 can send data to the camera system 100 and / or receive data from the camera system 100. In some embodiments, the communication device 156 can be implemented as a wireless communication module. In some embodiments, the communication device 156 can be implemented as a satellite connection to a proprietary system. In one example, the communication device 156 can be a hardwired data port (e.g., a USB port, a mini USB port, a USB-C connector, an HDMI port, an Ethernet port, a DisplayPort interface, a Lightning port, etc.). In another example, the communication device 156 can be a wireless data interface (e.g., Wi-Fi, Bluetooth, ZigBee, cellular, etc.).

[0048] The communication device 156 may be configured to receive the signal FEAT_SET. The signal FEAT_SET may include a feature set. The received feature set may be used to detect events and / or objects. For example, the feature set may be used to perform computer vision operations. The feature set information may include instructions for the processor 102 to determine which type of object corresponds to an object of interest and / or event.

[0049] The processor 102 may receive signals PIXELD_A-PIXELD_N from the capture devices 104a-104n at inputs 170a-170n. The processor 102 may send / receive signals (e.g., DATA) to / from the memory 160 at input / output 172. The processor 102 may receive signals SEN from the sensor 152 at input port 174. The processor 102 may receive signals DIR_AUD from the microphones 154a-154n at port 176. The processor 102 may send signals DIR_AOUT to the speakers 158a-158n via port 178. The processor 102 may send signals ENC_VIDEO to the communication device 156 via output port 180a. The processor 102 may receive signals FEAT_SET from the communication device 156 via input port 180b. In an example, the processor 102 may be connected to the capture devices 104a-104n, the sensor 152, the microphones 15a-154n, the communication device 156 and / or the speakers 158a-158n and / or the memory 160 via a bidirectional interface (or connection). The processor 102 may store and / or retrieve data from the memory 160. The memory 160 may be configured to store computer-readable / executable instructions (or firmware). The instructions, when executed by the processor 102, may perform a number of steps.

[0050] Signals PIXELD_A-PIXELD_N may include raw pixel data, thereby providing a field of view captured by lenses 106a-106n. Processor 102 may be configured to generate a video frame based on pixel data PIXELD_A-PIXELD_N. The video frame generated by processor 102 may be used within processor 102 (e.g., to perform video encoding, video transcoding, perform computer vision operations, etc.). In some embodiments, the video frame may be transferred to memory 160 for temporary storage. Processor 102 may be configured to generate an encoded video frame and transmit the encoded video frame to communication device 156 as signal ENC_VIDEO.

[0051] The processor 102 may be configured to make decisions based on analysis of video frames generated according to the signals PIXELD_A-PIXELD_N. The processor 102 may generate a signal ENC_VIDEO, a signal DATA, a signal DIR_AOUT, and / or other signals (not shown). The signal ENC_VIDEO, the signal DATA, and / or the signal DIR_AOUT may each be generated (in part) based on one or more decisions made and / or one or more functions performed by the processor 102. The decisions made and / or functions performed by the processor 102 may be determined based on data received by the processor 102 at inputs 170a-170n (e.g., signals PIXELD_A-PIXELD_N), at inputs 172, 174, 176, 180b, and / or other inputs.

[0052] Inputs 170a-170n, input / output 172, input 174, input 176, output 178, output 180a, input 180b, and / or other input / outputs may implement an interface. The interface may be implemented to transmit data to and / or from processor 102, sensor 152, communication device 156, capture device 104a-104n, memory 160, microphone 154a-154n, speaker 158a-158n, and / or other components of camera system 100. In one example, the interface may be configured to receive pixel data signals PIXELD_A-PIXELD_N from a respective one of capture devices 104a-104n (e.g., via input 170a-170n). In another example, the interface may be configured to receive sensor input from sensor 152 (e.g., via input 174). In yet another example, the interface may be configured to receive direction audio DIR_AUD (e.g., via input 176). In yet another example, the interface may be configured to send a coded video frame (e.g., signal ENC_VIDEO) and / or converted data determined based on a computer vision operation to the communication device 156. In another example, the interface may be configured to receive feature set information FEAT_SET from the communication device 156 (e.g., via input port 180b). In yet another example, the interface may be configured to send a directional audio output (e.g., signal DIR_AOUT) to each of the speakers 158a-158n. The interface may be configured to implement the transmission of data and / or convert data from one format to another format to ensure that the transmitted data is readable by the desired destination component. In an example, the interface may include a data bus, traces, connectors, wiring and / or pins. The implementation of the interface may vary according to the design criteria of a specific implementation.

[0053] Signal ENC_VIDEO may be presented to communication device 156. In some embodiments, signal ENC_VIDEO may include encoded video frames generated by processor 102. In some embodiments, the encoded video frames may include a full video stream (e.g., encoded video frames representing all videos captured by capture devices 104a-104n). In some embodiments, the encoded video frames may include time-lapse video. The encoded video frames may be encoded, clipped, spliced, and / or enhanced versions of the pixel data received from signals PIXELD_A-PIXELD_N. In an example, the encoded video frames may be high-definition, digital, encoded, de-distorted, stabilized, clipped, mixed, spliced, and / or rolling shutter effect corrected versions of signals PIXELD_A-PIXELD_N. In some embodiments, signal EN_VIDEO may include one or more disparity maps and / or stereoscopic images.

[0054] In some embodiments, the signal ENC_VIDEO may be generated based on video analysis (e.g., computer vision operations) and / or parallax operations performed by the processor 102 on video frames generated from the pixel data PIXELD_A-PIXELD_N. The processor 102 may be configured to perform computer vision operations to detect objects and / or events in the video frames, and then convert the detected objects and / or events into statistics and / or parameters. In one example, the data determined by the computer vision operations may be converted by the processor 102 into a human-readable format. The data from the computer vision operations may be used to detect objects and / or events to determine the video frames selected for the smart time-lapse video. The processor 102 may be configured to perform parallax operations to determine the distance of the camera system 100 from one or more detected objects.

[0055] In some embodiments, signal ENC_VIDEO may be data generated by processor 102 (e.g., video analysis results, audio / speech analysis results, etc.) that may be transmitted to a cloud computing service to aggregate information and / or provide training data for machine learning (e.g., to improve object detection, improve audio detection, etc.). In some embodiments, signal ENC_VIDEO may be provided to a cloud service for mass storage (e.g., to enable a user to retrieve the encoded video using a smartphone and / or desktop computer). The type of information transmitted by signal ENC_VIDEO may vary according to the design criteria of a particular implementation.

[0056] Circuit 100 may implement a camera system. In some embodiments, camera system 100 may be implemented as a drop-in solution (e.g., installed as a component). In an example, camera system 100 may be a device that may be installed as an aftermarket product (e.g., retro-fit for a drone, update for a security system, etc.). In some embodiments, camera system 100 may be a component of a security system. The number and / or type of signals and / or components implemented by camera system 100 may vary depending on the design criteria of a particular implementation.

[0057] The video data of the target vision captured by the capture devices 104a-104n can be generated according to the signals / bitstreams / data PIXELD_A-PIXELD_N. The capture devices 104a-104n can present the signals PIXELD_A-PIXELD_N to the inputs 170a-170n of the processor 102. The signals PIXELD_A-PIXELD_N can be used by the processor 102 to generate video frames / video data. In some embodiments, the signals PIXELD_A-PIXELD_N can be video streams captured by the capture devices 104a-104n. In some embodiments, the capture devices 104a-104n can be implemented in the camera system 100. In some embodiments, the capture devices 104a-104n can be configured to add existing functionality to the camera system 100.

[0058] Each of the capture devices 104a-104n may include a block (or circuit) 182, a block (or circuit) 184, and / or a block (or circuit) 186. The circuit 182 may implement a camera sensor (e.g., a complementary metal oxide semiconductor (CMOS) sensor). The circuit 184 may implement a camera processor / logic. The circuit 186 may implement a memory buffer. As a representative example, the capture device 104a is shown as including a sensor 182a, a logic block 184a, and a buffer 186a. Similarly, the capture devices 104b-104n may include camera sensors 182b-182n, logic blocks 184b-184n, and buffers 186b-186n. The sensors 182a-182n may each be configured to receive light from a corresponding one of the lenses 106a-106n and transform the light into digital data (e.g., a bit stream).

[0059] In one example, the sensor 182a of the capture device 104a may receive light from the lens 106a. The camera sensor 182a of the capture device 104a may perform a photoelectric conversion of the light from the lens 106a. In some embodiments, the sensor 182a may be an oversampled binary image sensor. In some embodiments, the camera sensor 182a may include an RGB sensor or an RGB-IR sensor. In some embodiments, the camera sensor 182a may include a rolling shutter sensor or a global shutter sensor. The logic 184a may convert a bit stream into human-perceivable content (e.g., pixel data and / or video data). For example, the logic 184a may receive pure (e.g., raw) data from the camera sensor 182a and generate pixel data (e.g., bit stream) based on the raw data. The memory buffer 186a may store raw data and / or processed bit streams. For example, the frame memory and / or buffer 186a may store one or more of the pixel data and / or video frames (e.g., video signals) (e.g., providing temporary storage and / or cache for them).

[0060] Microphones 154a-154n may be configured to capture incoming audio and / or provide directional information about incoming audio. Each of microphones 154a-154n may receive a corresponding signal (e.g., AIN_A-AIN_N). Signals AIN_A-AIN_N may be audio signals from an environment 50 near the camera system 100. For example, signals AIN_A-AIN_N may be external noise in the environment 50 and / or audio 164a-164n from objects 70a-70n. Microphones 154a-154n may be configured to generate signals DIR_AUD in response to signals AIN_A-AIN_N. Signal DIR_AUD may be a signal including audio data from signals AIN_A-AIN_N. Signal DIR_AUD may be a signal generated in a format that provides directional information about signals AIN_A-AIN_N.

[0061] The microphone may provide a signal DIR_AUD to the interface 176. The camera system 100 may include an interface 176 configured to receive data (e.g., signal DIR_AUD) from one or more of the microphones 154a-154n. In one example, data from the signal DIR_AUD presented to the interface 176 may be used by the processor 102 to determine the location of the source of the audio input. In another example, the microphones 154a-154n may be configured to determine the location of the audio input and present the location to the interface 176 as the signal DIR_AUD.

[0062] The number of microphones 154a-154n can vary according to the design criteria of a specific implementation. The number of microphones 154a-154n can be selected to provide sufficient directional information about the incoming audio (e.g., the number of microphones 154a-154n implemented can be based on the accuracy and / or resolution of the acquired directional information). In an example, 2 to 6 of the microphones 154a-156n can be implemented. In some embodiments, the microphones 154a-156n can be used to implement an audio processing component to process and / or encode the incoming audio signals AIN_A-AIN_N. In some embodiments, the processor 102 can be configured with on-chip audio processing to encode the incoming audio signals AIN_A-AIN_N. The microphones 154a-156n can capture the audio of the environment 50. The camera system 100 can be configured to synchronize the captured audio with the images captured by the capture devices 104a-104n.

[0063] The processor 102 may be configured to execute computer readable code and / or process information. The processor 102 may be configured to receive input and / or present output to the memory 160. The processor 102 may be configured to present and / or receive other signals (not shown). The number and / or type of inputs and / or outputs of the processor 102 may vary according to the design criteria of a particular implementation.

[0064] The processor 102 may receive signals PIXELD_A-PIXELD_N, signals SEN, signals DIR_AUD, signals FEAT_SET, and / or signals DATA. The processor 102 may make decisions based on data received at inputs 170a-170n, inputs 172, inputs 174, inputs 176, inputs 180b, and / or other inputs. For example, other inputs may include external signals generated in response to user inputs, external signals generated by sensors 152, microphones 154a-154n, and / or internally generated signals such as signals generated by the processor 102 in response to analysis of video frames and / or objects detected in video frames. The processor 102 may adjust (e.g., clip, digitally move, physically move camera sensors 182a-182n, etc.) video data of a video frame. Processor 102 may generate signal ENC_VIDEO and / or signal IDR_AOUT in response to data received by inputs 170a-170n, input 172, input 174, input 176, input 180b and / or decisions made in response to data received by inputs 170a-170n, input 172, input 174, input 176, and / or input 180b.

[0065] Signal ENC_VIDEO and / or signal DIR_AOUT may be generated to provide outputs in response to captured video frames, video encoding, and / or video analytics performed by processor 102. For example, video analytics may be performed by processor 102 in real time and / or near real time (e.g., with minimal latency).

[0066] The editing, downscaling, mixing, stabilization, packaging, encoding, compression, and / or conversion performed by processor 102 may vary according to the design criteria of a particular implementation. For example, the video frame generated by processor 102 may be a processed version of signals PIXELD_A-PIXELD_N configured to implement detection of an object and / or determination of characteristics of the detected object. In some embodiments, the video data may be encoded at a high bit rate. For example, the signal may be generated using lossless compression and / or with a small amount of loss.

[0067] In some embodiments, the video frames may be some views (or derivatives of some views) captured by the capture devices 104a-104n. For example, the video frames may include a portion of a panoramic video captured by the capture devices 104a-104n. In another example, the video frames may include a region of interest selected and / or clipped (e.g., upscaled, oversampled, and / or digitally zoomed) from the panoramic video frames by the processor 102 to achieve high-precision object detection. In some embodiments, the video frames may provide a series of clipped and / or enhanced video frames that improve the view from the perspective of the camera system 100 (e.g., provide night vision, provide high dynamic range (HDR) imaging, provide more visual areas, highlight detected objects, provide additional data such as numerical distance to detected objects, etc.) to enable the processor 102 to see better than a person can see using human vision.

[0068] The memory 160 may store data. The memory 160 may be implemented as a cache memory, a flash memory, a DRAM memory, etc. The type and / or size of the memory 160 may vary according to the design criteria of a particular implementation. The data stored in the memory 160 may correspond to video files of objects / events of interest, user profiles, user permissions, feature sets, types, information about the configuration of the lenses 106a-106n, etc.

[0069] The battery 162 may be configured to provide power to the components of the camera 100. The battery 162 may enable the camera 100 to operate without continuous access to an external power supply. In an example, the battery 162 may include a lithium-ion type battery. In another example, the battery 162 may include a supercapacitor. The type of battery 162 implemented may vary depending on the design criteria of a particular implementation.

[0070] In some embodiments, the lenses 106a-106n (e.g., camera lenses) can be oriented to provide a panoramic view from the camera system 100. The lenses 106a-106n can be intended to capture environmental data (e.g., light). The lenses 106a-106n can be configured to capture and / or focus light for the capture devices 104a-104n. Typically, the camera sensors 182a-182n can be located behind each of the corresponding lenses 106a-106n. Based on the light captured from the lenses 106a-106n, the capture devices 104a-104n can generate a bitstream and / or raw pixel data.

[0071] Embodiments of processor 102 may perform video stitching operations on signals PIXELD_A-PIXELD_N. In one example, each of pixel data signals PIXELD_A-PIXELD_N may provide a portion of a panoramic view, and processor 102 may clip, mix, synchronize and / or align pixel data from signals PIXELD_A-PIXELD_N to generate a panoramic video frame. In some embodiments, processor 102 may be configured to perform electronic image stabilization (EIS). Processor 102 may perform dewarping on video frames. Processor 102 may perform intelligent video analysis on dewarped video frames. Processor 102 discards video frames after video analysis and / or computer vision has been performed.

[0072] The encoded video frames may be processed locally. In one example, the encoded panoramic video may be stored locally by the memory 160 to enable the processor 102 to facilitate computer vision analysis internally (e.g., without first uploading the video frames to a cloud service). The processor 102 may be configured to perform operations such as computer vision operations and / or parallax operations locally (e.g., all operations are performed internally within the housing of the camera system 100 without using external resources such as a cloud service for processing).

[0073] The processor 102 may receive inputs to generate video frames (e.g., signals PIXELD_A-PIXELD_N) from the (multiple) CMOS sensors 182a-182n. The pixel data signals PIXELD_A-PIXELD_N may be enhanced by the processor 102 (e.g., color conversion, noise filtering, auto exposure, auto white balance, auto focus, etc.). Typically, a panoramic video may include a large field of view generated by one or more lenses / camera sensors. An example of a panoramic video may be an equirectangular 360 video. An equirectangular 360 video may also be referred to as a spherical panorama. A panoramic video may be a video that provides a larger field of view than may be displayed on a device for replaying the video. For example, the field of view captured by the camera system 200 may be used to generate panoramic videos such as spherical videos, hemispherical videos, 360 degree videos, wide angle videos, videos with a field of view less than 360 degrees, and the like.

[0074] The panoramic video may include a view of the environment near the camera system 100. In one example, the entire field of view of the panoramic video may generally be captured simultaneously (e.g., each portion of the panoramic video represents a view from the camera system 100 at a particular moment in time). In some embodiments (e.g., when the camera system 100 implements a rolling shutter sensor), there may be a small amount of time difference between portions of the panoramic video. Typically, each video frame of the panoramic video includes one exposure of the sensor (or multiple sensors 182a-182n) capturing the environment near the camera system 100.

[0075] In some embodiments, the field of view may provide coverage for a full 360 degree field of view. In some embodiments, a view less than 360 degrees may be captured by the camera system 100 (e.g., a 270 degree field of view, a 180 degree field of view, etc.). In some embodiments, the panoramic video may include a spherical field of view (e.g., video is captured above and below the camera system 100). For example, the camera system 100 may be mounted on a ceiling and capture a spherical field of view of an area below the camera system 100. In some embodiments, the panoramic video may include a field of view that is less than a spherical field of view (e.g., the camera system 100 may be configured to capture the ground below and areas to the sides of the camera system 100 rather than areas directly above). The implementation of the camera system 100 and / or the field of view captured may vary depending on the design criteria of a particular implementation.

[0076] In embodiments implementing multiple lenses, each of the lenses 106a-106n can be oriented in a particular direction to provide coverage for a full 360 degree field of view. In embodiments implementing a single wide angle lens (e.g., lens 106a), lens 106a can be positioned to provide coverage for a full 360 degree field of view (e.g., positioned on the bottom of the camera system 100 in a ceiling mounted embodiment, positioned on the bottom of a drone camera, etc.). In some embodiments, a view less than 360 degrees (e.g., a 270 degree field of view, a 180 degree field of view, etc.) can be captured by lenses 106a-106n. In some embodiments, lenses 106a-106n can be moved (e.g., the direction of the capture device can be controllable). In some embodiments, one or more of lenses 106a-106n can be configured to implement optical zoom (e.g., lenses 106a-106n can be zoomed in / out independently of each other).

[0077] In some embodiments, the camera system 100 may be implemented as a system on a chip (SoC). For example, the camera system 100 may be implemented as a printed circuit board including one or more components (e.g., capture devices 104a-104n, processor 102, communication device 156, memory 160, etc.). The camera system 100 may be configured to perform intelligent video analysis on video frames of a video. The camera system 100 may be configured to clip and / or enhance a video.

[0078] In some embodiments, the processor 102 may be configured to perform sensor fusion operations. The sensor fusion operations performed by the processor 102 may be configured to analyze information from multiple sources (e.g., capture devices 104a-104n, sensors 152, and microphones 154a-154n). By analyzing various data from different sources, the sensor fusion operations may be able to make inferences about data that may not be from one of the data sources alone. For example, the sensor fusion operations implemented by the processor 102 may analyze video data (e.g., people's mouth movements) and voice patterns from direction audio DIR_AUD. Different sources may be used to develop models of scenes to support decision making. For example, the processor 102 may be configured to compare the detected voice patterns with the synchronization of the mouth movements in the video frame to determine which person in the video frame is speaking. The sensor fusion operations may also provide time correlation, spatial correlation, and / or reliability among the received data.

[0079] In some embodiments, the processor 102 may implement convolutional neural network capabilities. The convolutional neural network capabilities may use deep learning techniques to implement computer vision. The convolutional neural network capabilities may be configured to implement pattern and / or image recognition using a training process through multiple layers of feature detection.

[0080] Signal DIR_AOUT may be an audio output. For example, processor 102 may generate output audio based on information extracted from video frames PIXELD_A-PIXELD_N. Signal DIR_AOUT may be determined based on events and / or objects determined using computer vision operations. In one example, signal DIR_AOUT may include an audio message of a detected person. In some embodiments, signal DIR_AOUT may not be generated until processor 102 has detected an event using computer vision operations.

[0081] Signal DIR_AOUT may include directional and / or positioning audio output information for speakers 158a-158n. Speakers 158a-158n may receive signal DIR_AOUT, process directional and / or positioning information and determine which speakers and / or which channels will replay a specific audio portion of signal DIR_AOUT. Speakers 158a-158n may generate signals AOUT_A-AOUT_N in response to signal DIR_AOUT. Signals AOUT_A-AOUT_N may be played audio messages. For example, speakers 158a-158n may send pre-recorded messages in response to detected events. Signal DIR_AOUT may be a signal generated in a format that provides directional information for signals AOUT_A-AOUT_N.

[0082] The number of speakers 158a-158n can vary according to the design criteria of a specific implementation. The number of speakers 158a-158n can be selected to provide sufficient directional channels for outgoing audio (e.g., the number of speakers 158a-158n implemented can be based on the accuracy and / or resolution changes of the directional audio output). In an example, 1 to 6 of the speakers 158a-158n can be implemented. In some embodiments, an audio processing component can be implemented by speakers 158a-158n to process and / or decode the output audio signal DIR_AOUT. In some embodiments, the processor 102 can be configured with on-chip audio processing. In some embodiments, the signal DIR_AOUT can replay the audio received from a remote device (e.g., a smart phone) to realize 2-way real-time audio communication.

[0083] The video pipeline of processor 102 can be configured to perform dewarping clipping, enhancement, rolling shutter correction, stabilization, downscaling, packaging, compression, conversion, mixing, synchronization and / or other video operations. The architecture of the video pipeline of processor 102 can implement real-time and / or near real-time video operations on high-resolution video and / or high-bitrate video data. The video pipeline of processor 102 can implement computer vision processing, stereoscopic vision processing, object detection, 3D noise reduction, fisheye lens correction (e.g., real-time 360-degree dewarping and lens distortion correction), oversampling and / or high dynamic range processing for 4K resolution video data. In one example, the architecture of the video pipeline can implement 4K ultra-high resolution encoded with H.264 at double real-time speed (e.g., 60fps), 4K ultra-high resolution and / or 4K AVC encoding using H.265 / HEVC at 30fps. The type of video operation and / or the type of video data operated by processor 102 can vary according to the design criteria of a specific implementation.

[0084] The sensors 182a-182n may each implement a high resolution sensor. Using the high resolution sensors 182a-182n, the processor 102 may combine oversampling of the image sensors 182a-182n with digital scaling within the clipping region. Oversampling and digital scaling may each be one of the video operations performed by the processor 102. Oversampling and digital scaling may be implemented to deliver a higher resolution image within the overall size constraints of the clipped region.

[0085] In some embodiments, one or more of the lenses 106a-106n may implement a fisheye lens. One of the video operations implemented by the processor 1012 may be a dewarping operation. The processor 102 may be configured to dewarp the generated video frames. Dewarping may be configured to reduce and / or remove sharp distortion caused by the fisheye lens and / or other lens characteristics. For example, dewarping may reduce and / or eliminate bulging effects to provide a straight image.

[0086] The processor 102 may be configured to clip (e.g., trim) a region of interest from a full video frame (e.g., generate a region of interest video frame). The processor 102 may generate a video frame and select a region. In an example, clipping the region of interest may generate a second image. The clipped image (e.g., the region of interest video frame) may generate a second image. The clipped image (e.g., the region of interest video frame) may be smaller than the original video frame (e.g., the clipped image may be a portion of a captured video).

[0087] The region of interest can be dynamically adjusted based on the location of the audio source. For example, the detected audio source can be mobile, and the location of the detected audio source can move as the video is captured. The processor 102 can update the selected region of interest coordinates and dynamically update the clipped portion (e.g., the directional microphones 154a-154n can dynamically update the position based on the captured directional audio). The clipped portion can correspond to the selected region of interest. As the region of interest changes, the clipped portion can also change. For example, the coordinates of the selected region of interest can change frame by frame, and the processor 102 can be configured to clip the selected region in each frame.

[0088] The processor 102 may be configured to oversample the image sensors 182a-182n. Oversampling of the image sensors 182a-182n may produce a higher resolution image. The processor 102 may be configured to digitally scale an area of ​​a video frame. For example, the processor 102 may digitally scale a clipped region of interest. For example, the processor 102 may establish a region of interest based on directional audio, clip the region of interest, and then digitally scale the clipped region of interest video frame.

[0089] The dewarping operation performed by the processor 102 may adjust the visual content of the video data. The adjustment performed by the processor 102 may make the visual content appear natural (e.g., as it would appear to a person viewing a location corresponding to the field of view of the capture device 104a-104n). In an example, the dewarping may alter the video data to generate a rectilinear video frame (e.g., to correct for artifacts caused by the lens characteristics of the lenses 106a-106n). The dewarping operation may be implemented to correct for distortion caused by the lenses 106a-106n. The adjusted visual content may be generated to achieve more accurate and / or reliable object detection.

[0090] Various features (dewarping, digital scaling, clipping, etc.) may be implemented as hardware modules in processor 102. Implementing hardware modules may increase the speed of video processing by processor 102 (e.g., faster than software implementations). Hardware implementations may enable processing of video while reducing the amount of latency. The hardware components used may vary depending on the design criteria of a particular implementation.

[0091] The processor 102 shown includes a plurality of blocks (or circuits) 190a-190n. Blocks 190a-190n may implement various hardware modules implemented by the processor 102. Hardware modules 190a-190n may be configured to provide various hardware components to implement a video processing pipeline. Circuits 190a-190n may be configured to receive pixel data PIXELD_A-PIXELD_N, generate video frames according to pixel data, perform various operations on video frames (e.g., de-warp, rolling shutter correction, cropping, magnification, image stabilization, etc.), prepare video frames to communicate with external hardware (e.g., encoding, packaging, color correction, etc.), parse feature sets, implement various operations of computer vision, etc. Various implementations of the processor 102 may not necessarily utilize all features of the hardware modules 190a-190n. The features and / or functions of the hardware modules 190a-190n may vary according to the design criteria of a specific implementation. Details of hardware modules 190a-190n may be described in association with U.S. patent application Ser. No. 16 / 831,549 filed on April 16, 2020, U.S. patent application Ser. No. 16 / 288,922 filed on February 28, 2019, and U.S. patent application Ser. No. 15 / 593,493 filed on May 12, 2017 (now U.S. Pat. No. 10,437,600), appropriate portions of which are incorporated herein by reference in their entirety.

[0092] Hardware modules 190a-190n can be implemented as dedicated hardware modules. Compared with software implementation, using dedicated hardware modules 190a-190n to implement various functions of processor 102 can enable processor 102 to be highly optimized and / or customized to limit power consumption, reduce heat generation and / or increase processing speed. Hardware modules 190a-190n can be customizable and / or programmable to implement various types of operations. Implementing dedicated hardware modules 190a-190n can enable the hardware used to perform each type of calculation to be optimized for speed and / or efficiency. For example, hardware modules 190a-190n can implement multiple relatively simple operations frequently used in computer vision operations, and these operations can enable computer vision algorithms to be executed in real time together. Video pipelines can be configured to identify objects. Objects can be identified by interpreting digital and / or symbolic information to determine that visual data represents a specific type and / or feature of an object. For example, the number of pixels of video data and / or the color of the pixels can be used to identify a portion of the video data as an object. The hardware modules 190a - 190n may enable computationally intensive operations (eg, computer vision operations, video encoding, video transcoding, etc.) to be performed locally on the camera 100 .

[0093] One of the hardware modules 190a-190n (e.g., 190a) can implement a scheduler circuit. Scheduler circuit 190a can be configured to store a directed acyclic graph (DAG). In an example, scheduler circuit 190a can be configured to generate and store a directed acyclic graph in response to feature set information received in signal FEAT_SET. A directed acyclic graph can define video operations to be performed to extract data from a video frame. For example, a directed acyclic graph can define various mathematical weights (e.g., neural network weights and / or biases) to be applied when performing computer vision operations to classify various pixel groups into specific objects.

[0094] The scheduler circuit 190a may be configured to parse the acyclic graph to generate various operators. Operators may be scheduled by the scheduler circuit 190a in one or more of the other hardware modules 190a-190n. For example, one or more of the hardware modules 190a-190n may implement a hardware engine configured to perform a specific task (e.g., a hardware engine designed to perform specific mathematical operations repeatedly used to perform computer vision operations). The scheduler circuit 190a may schedule operators based on when the operators are ready to be processed by the hardware engines 190a-190n.

[0095] The scheduler circuit 190a can time-multiplex tasks to the hardware modules 190a-190n to perform work based on the availability of the hardware modules 190a-190n. The scheduler circuit 190a can parse the directed acyclic graph into one or more data streams. Each data stream can include one or more operators. Once the directed acyclic graph is parsed, the scheduler circuit 190a can assign the data stream / operator to the hardware engine 190a-190n and send the relevant operator configuration information to start the operator.

[0096] The binary representation of each directed acyclic graph can be an ordered traversal of the directed acyclic graph, where descriptors and operators are interleaved based on data dependencies. Descriptors typically provide registers that link data buffers to specific operands in dependent operators. In various embodiments, an operator may not appear in the directed acyclic graph representation until all dependent descriptors have been declared for the operands.

[0097] One of the hardware modules 190a-190n (e.g., 190b) may implement a convolutional neural network (CNN) module. The CNN module 190b may be configured to perform computer vision operations on video frames. The CNN module 190b may be configured to implement object and / or event recognition through multi-layer feature detection. The CNN module 190b may be configured to calculate descriptors based on the feature detection performed. The descriptors may enable the processor 102 to determine the likelihood that a pixel of a video frame corresponds to a particular object (e.g., a person, a pet, an item, text, etc.).

[0098] The CNN module 190b may be configured to implement convolutional neural network capabilities. The CNN module 190b may be configured to implement computer vision using deep learning techniques. The CNN module 190b may be configured to implement pattern and / or image recognition using a training process through multi-layer feature detection. The CNN module 190b may be configured to perform inference for a machine learning model.

[0099] CNN module 190b can be configured to perform feature extraction and / or matching only in hardware. Feature points typically represent areas of interest (e.g., corners, edges, etc.) in a video frame. By temporarily tracking feature points, an estimate of the self-motion of the capture platform or a motion model of an object observed in the scene can be generated. In order to track feature points, a matching algorithm is usually incorporated into CNN module 190b by hardware to find the most likely correspondence between feature points in a reference video frame and a target video frame. In the process of matching reference feature points and target feature points, each feature point can be represented by a descriptor (e.g., image block, SIFT, BRIEF, ORB, FREAK, etc.). Using dedicated hardware circuits to implement CNN module 190b can achieve real-time calculation of descriptor matching distances.

[0100] CNN module 190b can be a dedicated hardware module configured to perform feature detection of video frames. Features detected by CNN module 190b can be used to calculate descriptors. CNN module 190b can determine the possibility that pixels in a video frame belong to a specific object and / or multiple objects in response to descriptors. For example, using descriptors, CNN module 190b can determine the possibility that a pixel corresponds to a specific object (e.g., a person, a piece of furniture, a photo of a person, a pet, etc.) and / or a feature of an object (e.g., a person's mouth, a person's hand, a screen of a television, an armrest of a sofa, a clock, etc.). Implementing CNN module 190b as a dedicated hardware module of processor 102 can enable camera system 100 to perform computer vision operations locally (e.g., on-chip) without relying on the processing power of a remote device (e.g., transmitting data to a cloud computing service).

[0101] The computer vision operations performed by the CNN module 190b can be configured to perform feature detection on the video frame to generate a descriptor. The CNN module 190b can perform object detection to determine the area in the video frame with a high probability of matching a specific object. In one example, an open operand stack can be used to customize the type of object to be matched (e.g., a reference object) (thereby enabling the programmability of the processor 102 to implement various directed acyclic graphs, each directed acyclic graph providing instructions for performing various types of object detection). The CNN module 190b can be configured to perform local masking to detect objects in areas with a high probability of matching (multiple) specific objects.

[0102] In some embodiments, the CNN module 190b can determine the location (e.g., 3D coordinates and / or position coordinates) of various features (e.g., characteristics) of the detected object. In one example, the 3D coordinates can be used to determine the position of the arms, legs, chest, and / or eyes. A position coordinate for the vertical position of the body part in 3D space on the first axis and another coordinate for the horizontal position of the body part in 3D space on the second axis can be stored. In some embodiments, the distance from the lens 106a-106n can represent a coordinate for the depth position of the body part in 3D space (e.g., a position coordinate on the third axis). Using the position of each body part in 3D space, the processor 102 can determine the body location and / or physical characteristics of the person 70a-70n.

[0103] The CNN module 190b may be pre-trained (e.g., the CNN module 190b may be configured to perform computer vision based on received training data to detect an object to train the CNN module 190b). For example, the results of the training data (e.g., a machine learning model) may be pre-programmed and / or loaded into the processor 102. The CNN module 190b may perform inferences for the machine learning model (e.g., to perform object detection). Training may include determining weighted values ​​(e.g., neural network weights) for each of a plurality of layers. For example, a weighted value may be determined for each of a layer for feature extraction (e.g., a convolutional layer) and / or a layer for classification (e.g., a fully connected layer). The weighted values ​​learned by the CNN module 190b may vary according to the design criteria of a particular implementation.

[0104] The convolution operation may include sliding a feature detection window along the layer while performing calculations (e.g., matrix operations). The feature detection window may apply filters to pixels and / or extract features associated with each layer. The feature detection window may be applied to a pixel and a plurality of surrounding pixels. In an example, a layer may be represented as a matrix representing the values ​​of a pixel and / or a feature of one of the layers, and the filter applied by the feature detection window may be represented as a matrix. The convolution operation may apply matrix multiplication between regions of the current layer covered by the feature detection window. The convolution operation may slide the feature detection window along the region of the layer to generate a result representing each region. The size of the region, the type of operation applied by the filter, and / or the number of layers may vary according to the design criteria of a particular implementation.

[0105] Using convolution operations, CNN module 190b can calculate multiple features for pixels of the input image in each extraction step. For example, each of the layers can receive input from a feature set in a small neighborhood (e.g., region) located in a previous layer (e.g., local receptive field). Convolution operations can extract basic visual features (e.g., oriented edges, endpoints, corners, etc.), which are then combined by higher layers. Since the feature extraction window operates on pixels and nearby pixels (or sub-pixels), the result of the operation can have position invariance. The layer may include a convolution layer, a pooling layer, a nonlinear layer, and / or a fully connected layer. In an example, a convolution operation can learn to detect edges from raw pixels (e.g., the first layer), then use features from the previous layer (e.g., detected edges) to detect shapes in the next layer, and then use the shapes to detect higher-level features in the higher layer (e.g., facial features, pets, furniture, etc.), and the last layer can be a classifier using higher-level features.

[0106] CNN module 190b can execute data flows directed to feature extraction and matching, including two-stage detection, deformation operators, component operators that manipulate lists of components (e.g., components can be regions of vectors that share common attributes and can be grouped together with bounding boxes), matrix inversion operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, minimum and maximum reduction operators, pooling operators, non-minimum non-maximum suppression operators, non-maximum suppression operators based on scanning windows, aggregation operators, scattering operators, statistical operators, classifier operators, integral image operators, comparison operators, indexing operators, pattern matching operators, feature extraction operators, feature detection operators, two-stage object detection operators, score generation operators, block reduction operators, and upsampling operators. The types of operations performed by CNN module 190b to extract features from training data can vary according to the design criteria of a particular implementation.

[0107] The hardware module 190c may be configured to implement a parallax engine. The parallax engine 190c may be configured to perform parallax operations. The parallax engine 190c may be configured to determine distances based on images captured as a stereo pair. Two or more of the capture devices 104a-104n may be configured as a stereo pair of cameras (e.g., a stereo camera). The capture devices 104a-104n configured as a stereo pair may be implemented to be close to each other at a predefined distance and / or have a symmetrical orientation about a center position. The capture devices 104a-104n configured as a stereo pair may be configured to capture video frames from similar but slightly different angles (e.g., at an inner angle to capture overlapping fields of view).

[0108] The parallax engine 190c may be configured to perform a comparison to analyze the differences between the stereo pairs of images. In an example, the processor 102 may detect feature points of the same object detected in two video frames (e.g., a reference image and a target image) captured by the capture devices 104a-104n configured as a stereo pair. The parallax engine 190c may determine the distance (e.g., offset) of the feature points and then determine the calculation based on the characteristics of the stereo pair of capture devices (e.g., angle, distance of separation, etc.) and the determined distance of the feature points. Based on the differences between the stereo pairs of images and the predefined distances between the capture devices 104a-104n configured as a stereo pair, the parallax engine 190c may be configured to determine the distance. The distance determined by the parallax engine 190c may be the distance from the capture devices 104a-104n configured as a stereo pair. In an example, the disparity engine 190c may determine the distance to a particular object (e.g., a vehicle, bicycle, pedestrian, driver, vehicle occupant, etc.) from the capture devices 104a-104n configured as a stereo pair based on comparing disparities in a stereo pair of captured images.

[0109] The predetermined distance and predetermined angle of the lenses 106a-106b may be used by the disparity engine 190c to generate disparity values, disparity maps, and / or stereoscopic images. In some embodiments, the disparity engine 190c may implement a cache memory for storing calibration data about the lenses 106a-106b of the stereo camera 100 and / or other capture devices 104a-104n.

[0110] The disparity engine 190c may be configured to compare pixels of a reference image and a target image. The disparity engine 190c may determine a match score between positions in the reference image and the target image. The match score may provide an indication of how similar a pixel at one position in the target image is to a pixel at another position in the reference image. The match score may be used to determine an offset between the target image and the reference image. The disparity engine 190c may be configured to determine a match score and / or offset value using block matching techniques / calculations, semi-global matching techniques / calculations, implementing guide nodes, etc. The disparity engine 190c may be configured to determine a peak offset position for a single pixel in the target image and the reference image. The peak offset position may be determined at a sub-pixel level.

[0111] Each of the hardware modules 190a-190n may implement a processing resource (or hardware resource or hardware engine). The hardware engines 190a-190n may be operable to perform specific processing tasks. In some configurations, the hardware engines 190a-190n may operate in parallel and independently of each other. In other configurations, the hardware engines 190a-190n may operate in conjunction with each other to perform assigned tasks. One or more of the hardware engines 190a-190n may be a homogeneous processing resource (all circuits 190a-190n may have the same capabilities) or a heterogeneous processing resource (two or more circuits 190a-190n may have different capabilities).

[0112] refer to Figure 4 , a diagram illustrating the parallax between a reference image and a target image is shown. Example video frames 200a-200b are shown. Video frames 200a-200b may include a stereo pair of images captured by a stereo camera 100. In an example, video frame 200a may be a video frame generated based on pixel data (e.g., pixel data PIXELD_A) captured by a right capture device 104a, and video frame 200b may be a video frame generated based on pixel data (e.g., pixel data PIXELD_B) captured by a left capture device 104b. In the example shown, the stereo pair 200a-200b of the image may correspond to the horizontal orientation of the lens 106a-106b. However, the description of the stereo pair 200a-200b of the image may be applicable to any orientation of the stereo pair 106a-106b of the lens. In one example, image 200a may be a reference image and image 200b may be a target image. In another example, image 200b may be a reference image and image 200a may be a target image. Which image of the stereoscopic pair of images is used by disparity engine 190c as the target image and the reference image may vary according to the design criteria of a particular implementation.

[0113] The stereo pair of example video frames 200a-200b may illustrate a portion of video data captured by capture device 104a and / or capture device 104b (e.g., the field of view captured by capture device 104a may include more video data than shown in video frame 200a and / or the field of view captured by capture device 104b may include more video data than shown in video frame 200b). The stereo pair of video frames 200a-200b may be a representative example of video frames captured by a virtual stereo pair of cameras implemented by apparatus 100.

[0114] The stereo pair 200a-200b of video frames are shown as rectilinear video frames. The rectilinear processing pipeline of the processor 102 may include a dewarping engine configured to perform dewarping video operations to generate rectilinear video frames based on one or more non-rectilinear video frames captured by the capture devices 104a-104n. In an example, the processor 102 may receive a non-rectilinear video frame having barrel / fisheye lens characteristics (e.g., captured by a camera with a 160 degree field of view lens). In some embodiments, the processor 102 may be configured to perform video operations and / or comparison operations (e.g., detecting objects, determining distances, determining relative speeds, etc.) on the non-rectilinear video frames presented to the processor 102. In some embodiments, the processor 102 may be configured to dewarp the non-rectilinear video frames using the dewarping engine and then perform video operations on the rectilinear video frames.

[0115] The stereo pair of video frames 200a-200b may include video data captured from similar but slightly different angles. Typically, the stereo pair of video frames 200a-200b may capture the same area and / or object. Capturing similar views of the area and / or object in the stereo pair of video frames 200a-200b may enable the parallax engine 190c to perform calculations (e.g., comparison operations) to determine the relative distance and / or speed of the detected object.

[0116] Video frame 200a may include video data of vehicle 202 and a portion of vehicle 204. A dashed shape 210 is shown. Dashed shape 210 may represent detection of an object by a computer vision operation performed by processor 102. Dashed shape 210 may include pixel data corresponding to an object detected by a computer vision operation pipeline and / or CNN module 190b. Dashed shape 210 is shown for illustrative purposes. In an example, dashed shape 210 may be a visual representation of object detection (e.g., dashed shape 210 may not appear on an output video frame). In another example, dashed shape 210 may be a bounding box generated by processor 102 that is displayed on an output video frame to indicate that an object has been detected (e.g., bounding box 210 may be displayed in a debug mode of operation).

[0117] Computer vision operations can be configured to detect the characteristics of detected objects and / or the behavior of detected objects. The characteristics of an object can include height, length, width, slope, arc length, color, the amount of light emitted, text on the detected object, moving path, moving speed, moving direction, etc. The characteristics of the detected object can include the state of the object (e.g., open, closed, on, off, etc.). The behavior can be determined in response to the type of object and / or the characteristics of the detected object. Although a stereo pair 200a-200b of an example video frame is shown, the behavior of the object can be determined by analyzing the sequence of video frames captured over time. For example, the moving path and / or moving speed characteristics can be used to determine that an object classified as a person may be walking or running. The type of characteristics and / or the detected behavior can vary according to the design criteria of a specific implementation.

[0118] Vehicle 202 is shown as being detected as object 210. A plurality of dashed circles 212a-212n are shown. Dashed circles 212a-212n may represent feature points of object 210 detected by processor 102 (e.g., CNN module 190b and / or disparity engine 190c). In an example, feature point 212a may correspond to the upper left corner of vehicle 202, feature point 212b may correspond to the driver's side mirror of vehicle 202, feature point 212c may correspond to the hood of vehicle 202, feature point 212d may correspond to the left rear of vehicle 202, and feature point 212n may correspond to the right front tire of vehicle 202. The location, type, and / or amount of feature points 212a-212n detected by processor 202 may vary according to the design criteria of a particular implementation.

[0119] Video frame 200b may include vehicle 202 and portion 206 of vehicle. Vehicle 202 is shown as being detected as object 210 (e.g., the same object as in video frame 200a). A plurality of dashed circles 214a-214n are shown. Dashed circles 214a-214n may represent feature points of object 210 detected by processor 102. In an example, feature point 214a may correspond to the upper left corner of vehicle 202, feature point 214b may correspond to the driver's side mirror of vehicle 202, feature point 214c may correspond to the hood of vehicle 202, feature point 214d may correspond to the left rear of vehicle 202, and feature point 214n may correspond to the right front tire of vehicle 202. In general, feature points 214a-214n in video frame 200b may correspond to corresponding feature points 212a-212n detected in video frame 200a, but may be shifted due to an offset between lenses 106a-106b (eg, horizontal shift for a horizontally oriented stereo camera).

[0120] Vehicle 202 is shown as being located in a slightly different positioning in each of the stereo pairs 200a-200b of the video frames. Vehicle 206 is partially shown in video frame 200b but not in video frame 200a. Vehicle 204 is partially shown in video frame 200a but not in video frame 200b. The different positions of vehicles 202-206 in the stereo pairs 200a-200b of the video frames can represent the stereo pairs 200a-200b of the video frames that have been captured from slightly different angles. For example, video frame 200a can be captured at a distance corresponding to the offset between the video frames 200a-200b and the lenses 106a-106b. The stereo pairs 200a-200b of the video frames can capture different angles. For example, capture device 104a and capture device 104b can be at an inner angle (e.g., instead of the target being straight outward from stereo camera 100). Because capture device 104a and capture device 104b may be at an inner angle, the fields of view captured by both lenses 106a-106b may capture overlapping areas. The offset between the stereo pair of images 200a-200b may be exaggerated for illustrative purposes. The offset between the stereo pair of images 200a-200b may correspond to the distance between the stereo pair of lenses 106a-106b and the distance of the object (e.g., vehicles 202-206) from the lenses 106a-106b (e.g., the inner angle of the stereo pair of lenses 106a-106b may cause objects closer to the stereo camera 100 to appear to have more offset).

[0121] The CNN module 190b may be configured to detect an object 210 in a stereo pair 200a-200b of video frames. The CNN module 190b may be configured to identify a vehicle 202 as the same object in both stereo pairs 200a-200b of video frames. The CNN module 190b may detect feature points 212a-212n in the video frame 200a and cross-reference feature points 214a-214n detected in the video frame 200b. The CNN module 190b may compare the feature points 212a-212n with the feature points 214a-214n to determine that the vehicle 202 detected as the object 210 in the video frame 200a is the same as the vehicle 202 detected as the object 210 in the video frame 200b. The CNN module 190b may generate a confidence level to indicate the likelihood that the object 210 is the same object in both stereo pairs 200a-200b of video frames. If the confidence level is above a predetermined threshold, the processor 102 may determine that the object 210 is in both stereoscopic video frames 200a-200b and enable the disparity engine 190c to perform the comparison.

[0122] If the CNN module 190b determines that the object 210 is the same object in both of the stereo pairs 200a-200b of the video frames, the disparity engine 190c can determine the relative distance and / or relative speed of the object 210 relative to the stereo camera 100. The disparity engine 190c can be configured to determine the relative distance and / or speed of the object 210 based on the different angles and symmetric orientations of the stereo pairs 104a-104b of the capture devices. The disparity engine 190c can be configured to perform a comparison operation by calculating the difference (e.g., disparity) between the object 210 in the video frame 200a and the object 210 captured in the video frame 200b. The disparity engine 190c can determine the difference by comparing various feature points of the object 210 detected by the CNN module 190b.

[0123] Dashed lines 220a-220n are shown. Dashed lines 220a-220n may connect between feature points 212a-212n of video frame 200a and corresponding feature points 214a-214n of video frame 200b, respectively. In the example shown, dashed line 220a may connect feature point 212a to feature point 214a, dashed line 220b may connect feature point 212b to feature point 214b, dashed line 220c may connect feature point 212c to feature point 214c, dashed line 220d may connect feature point 212d to feature point 214d, and dashed line 220n may connect feature point 212n to feature point 214n. Dashed lines 220a-220n may connect to the same feature of object 210 in both stereo pairs 200a-200n of video frames.

[0124] Dashed lines 220a-220n may represent a comparison operation performed by the disparity engine 190c. The disparity engine 190c may perform a comparison operation by determining an offset distance between feature points 212a-212n of the video frame 200a and corresponding feature points 214a-214n of the video frame 200b. Dashed lines 220a-220n may be a visual representation of the comparison performed by the disparity engine 190c. The disparity engine 190c may measure the distance (e.g., horizontal distance in the illustrated example) between the locations of the feature points 212a-212n and the feature points 214a-214n.

[0125] The parallax engine 190c may be configured to perform geometric and / or trigonometric calculations based on the measured distances between the feature points 212a-212n and the feature points 214a-214n and the characteristics of the stereo pair 104a-104b of the capture device. The calculations performed by the parallax engine 190c may determine the relative distance of the object 210 from the stereo pair 104a-104b of the capture device. The calculations performed by the parallax engine 190c may determine the relative speed of the object 210 relative to the stereo camera 100 by comparing the stereo pair 200a-200b of images captured over time (e.g., by analyzing a sequence of stereo pairs of images and comparing changes in the relative distance between the object 210 and the stereo pair 104a-104b of the capture device over time as the images are captured). The details of the parallax operations performed by the parallax engine 190c may be described in detail in detail in conjunction with the accompanying drawings. Figure 5 Describe in association.

[0126] The parallax engine 190c can determine the distance of the object 210. By knowing the distance, the parallax engine 190c can determine (e.g., measure) the physical width of the object 210. The width W is shown above the object 210 in the video frame 200a. The width W can represent the physical width of the vehicle 202. In an example, the parallax engine 190c can convert the number of pixels of the object 210 into a width measurement (e.g., feet, inches, meters, etc.) based on the distance determined using the comparison operation and / or a predetermined distance from the stereo camera 100. In an example, the conversion of the number of pixels to the width measurement can be determined using geometric and / or trigonometric calculations based on the measured distances between the feature points 212a-212n and the feature points 214a-214n and the characteristics of the stereo pair 104a-104b of the capture device.

[0127] refer to Figure 5 , a diagram illustrating the use of a guide node to generate disparity values ​​is shown. An example representation of a disparity calculation 250 is shown. The example representation of a disparity calculation 250 may include images 252a-252b, a graph 254, and a portion of a graph 256. The example portion of images 252a-252b may be a portion of the example stereoscopic images 200a-200b. In an example, portion 252a may be a portion of Figure 4 The portion of the reference image 200b shown in association with FIG. 200b and the portion 252b may be associated with Figure 4 Portions of target image 200a are shown in association. Portions 252a-252b are shown for purposes of illustrating the disparity calculations performed by disparity engine 190c. Typically, disparity calculations are performed by disparity engine 190c for the entire stereoscopic images 200a-200b.

[0128] The target image portion 252a may be included in Figure 4200b) is associated with the video data of the vehicle 202 (e.g., the object 210) shown in the video frame 200b (e.g., the front grille of the vehicle is shown as an example). The target image 252a is shown to include dashed lines 260a-260b and dashed lines 262a-262n. The dashed lines 260a-260b can be horizontal lines across the target image 252a, and the dashed lines 262a-262n can be vertical lines across the target image 252a. The horizontal lines 260a-260n and the vertical lines 262a-262n can form a grid pattern across the target image 252a. The grid pattern formed by the horizontal lines 260a-260n and the vertical lines 262a-262n can illustrate the boundaries of the guide nodes used for the disparity calculations performed by the disparity engine 190c.

[0129] Pixel 270 is shown. Pixel 270 may be the center pixel of the guide node. Center pixel 270 may be the guide node pixel. Disparity engine 190c may be configured to perform a disparity operation over the full search range to determine a peak matching position for center pixel 270 of the guide node.

[0130] A dashed box 272 is shown. The dashed box 272 may be located around the center pixel 270. The dashed box 272 may include a pixel block. The pixel block 272 may include the center pixel 270 and a plurality of nearby pixels (e.g., a center pixel block). In some embodiments, the center pixel block 272 may include only pixels adjacent to the center pixel 270. In some embodiments, the center pixel block 272 may include pixels adjacent to the center pixel 270 and other pixels near the center pixel 270. The parallax engine 190c may be configured to clip the center pixel block 272 from the target image 252a to perform a full range search. The number of pixels in the center pixel block 272 may vary according to the design criteria of a specific implementation.

[0131] Pixel 274 is shown. Pixel 274 can be a surrounding pixel. Surrounding pixel 274 can be another pixel within the guiding node including central pixel 270 (e.g., a pixel other than central pixel 270). For example, surrounding pixel 274 can be located near central pixel 270 within the same guiding node as central pixel 270. In one example, surrounding pixel 274 can be a pixel within central pixel block 272. In another example, surrounding pixel 274 can be a pixel within central pixel block 272. In another example, surrounding pixel 274 can be outside central pixel block 272 but still located within the guiding node of central pixel 270. In the example shown, one surrounding pixel 274 is marked. However, a guiding node with central pixel 270 can include more than one surrounding pixel (e.g., all pixels within the guiding node except central pixel 270 can be one of the surrounding pixels).

[0132] A dashed box 276 is shown. The dashed box 276 may be located around the surrounding pixels 274. The dashed box 276 may include a pixel block. The pixel block 276 may include the surrounding pixels 274 and a plurality of nearby pixels (e.g., a surrounding pixel block). In some embodiments, the surrounding pixel block 276 may include only pixels adjacent to the surrounding pixels 274. In some embodiments, the surrounding pixel block 276 may include pixels adjacent to the surrounding pixels 274 and other pixels near the surrounding pixels 274. Depending on the distance of the surrounding pixels 274 from the center pixel 270, the surrounding pixel block 276 may include the center pixel 270 or may not include the center pixel 270. The parallax engine 190c may be configured to clip the surrounding pixel block 276 from the target image 252a to perform a search in a local range. The number of pixels in the surrounding pixel block 276 may vary according to the design criteria of a specific implementation.

[0133] Positions 278a-278g are shown. Positions 278a-278g can represent the boundaries of guide nodes. Guide nodes 278a-278g can include central pixel 270 and surrounding pixels of central pixel 270 (e.g., surrounding pixels 274 shown and other pixels near central pixel 270 within the boundaries of guide nodes 278a-278g). Guide nodes 278a-278g can be selected along a grid pattern including horizontal lines 260a-260n and vertical lines 262a-262n. For example, the parallax engine 190c can be configured to select guide nodes 278a-278g with a size (e.g., X multiplied by the size of Y) that runs through the position on the reference image 200a.

[0134] In an example, the parallax engine 190c can select the position for the guide node 278a-278g, perform the parallax operation, then select the new position for the guide node 278a-278g at the next position (e.g., adjacent position) and perform the parallax operation again. The next position for the guide node 278a-278g can be a position that does not overlap with the previous position of the guide node 278a-278g. The movement of the guide node 278a-278g can be repeated until the parallax operation has been performed for each pixel of the stereo pair 200a-200b of the image. In an example, the position of the guide node 278a-278g can slide (or shift) along the grid lines 260a-260n and 262a-262n. The guide node 278a-278g can be shifted vertically or horizontally. In some embodiments, the guide node 278a-278g can move to a position that is not adjacent to the previous position of the guide node. The size and / or movement of the guide nodes 278a-278g may be selected for efficient processing of the stereoscopic pair of images 200a-200n (e.g., based on the available portions 252a-252b loaded in the cache memory). The size and / or movement of the guide nodes 278a-278g may vary according to the design criteria of a particular implementation.

[0135] A dashed box 280 is shown. The dashed box 280 may be a search range. The search range 280 may be a representation of a full search range. The full search range 280 may be a set of locations where the disparity engine 190c performs a disparity operation for the center pixel 270. For example, the disparity engine 190c may perform a comparison operation between the target image 200b and the reference image 200a to determine the most likely matching location for the center pixel 270. The comparison operation may include performing a block match of the center pixel block 272 at multiple locations within the full search range 280 to generate a matching score. When the BM technique is implemented, the full search range 280 may be similar to a search range for all pixels.

[0136] A dashed box 282 is shown. The dashed box 282 may represent a guide node pixel peak position. The guide node pixel peak position 282 may be a position with a highest match score within the full search range, the highest match score being generated in response to performing a block matching operation between the target image 252a and the reference image 252b. In an example, the disparity engine 190c may determine a shift offset (e.g., a disparity value) for the center pixel 270 based on the guide node pixel peak position 282. The guide pixel peak position 282 may be used as a disparity value for a disparity map.

[0137] A dashed box 284 is shown. The dashed box 284 can be a search range. The search range 284 can be a representation of a local search range. The local search range 284 can be a set of positions where the disparity engine 190c performs a disparity operation on the surrounding pixels of the center pixel 270 within the guide node 278a-278g. For example, the disparity engine 190c can perform a comparison operation between the target image 200b and the reference image 200a to determine the most likely matching position for the surrounding pixels 274 (and other surrounding pixels). The comparison operation can include performing a block match of the surrounding pixel block 276 at multiple positions within the local search range 284 to generate a matching score.

[0138] The local range 284 may be smaller than the full search range 280. The number of locations where the disparity engine 190c performs a disparity operation (e.g., block matching) may be less in the local search range 284 for surrounding pixels than in the full search range 280 for the center pixel 270. In one example, the disparity engine 190c may perform block matching operations at three locations within the local search range 284.

[0139] In the example shown, the local search range 284 may correspond to the surrounding pixels 274. The disparity engine 190c may determine the matching scores at the positions in the local search range 284 (e.g., generate three matching scores). The matching scores may be generated in response to performing a block matching operation between the target image 252a and the reference image 252b within the local search range 284. The disparity engine 190c may determine the shift offset peak position (e.g., disparity value) for the surrounding pixels 274 based on calculating a parabola defined by the matching scores determined at the positions within the local search range 284. The shift offset peak position may be used as the disparity value for the surrounding pixels. In the example shown, one local search range 284 corresponding to the surrounding pixels 274 is shown. The disparity engine 190c may select a local search range 284 for each of the surrounding pixels in the guide nodes 178a-278g to determine the shift offset peak position.

[0140] A dashed box 290 is shown on the target image portion 252b. The dashed box 290 may be a pixel on the target image portion 252b that corresponds to the center pixel 270 on the reference image portion 252a (e.g., a matching pixel). Figure 2 In some embodiments, matching pixel 290 may be offset (e.g., horizontally shifted) from center pixel 270. In some embodiments, matching pixel 290 may be vertically offset (e.g., horizontally shifted) from center pixel 270. Figure 1200a and target image 200b. For example, matching pixel 290 may be aligned with guide pixel peak position 282 in full search range 280 (e.g., if reference image portion 252a is overlaid on top of target image portion 252b). Matching pixel 290 may be offset due to the distance and / or angle between stereo lenses 106a-106b. The offset between center pixel 270 and matching pixel 290 may vary at different locations in reference image 200a and target image 200b. For example, the difference in offset may be due to the angle between lenses 106a-106b, vibration of lenses 106a-106b, distance of an object from lenses 106a-106b, etc.

[0141] The graph 254 may include a Y-axis 292 and an X-axis 294. The Y-axis 292 may represent the match score values ​​determined by the disparity engine 190c for the center pixel block 272 at various locations in the full search range 280. The X-axis 294 may represent the horizontal (e.g., X) pixel locations of the reference image 252a and the target image 252b. For example, the X-axis 294 may represent where the match score is calculated in the stereo pair 252a-252b of images.

[0142] A curve 296 is shown. Curve 296 can represent the match score values ​​generated by the disparity engine 190c along the full search range 280. Curve 296 is shown as generally having the same width as the full search range 280. Curve 296 shows generally lower match scores on the left and far right. Point 298 is shown on curve 296. Point 298 can represent the guide pixel peak position. Line 300 is shown extending from the guide pixel peak position 298. Line 300 can be aligned with the position of the matching pixel 290 in the target image portion 252b and the guide pixel peak position 282 shown in the full search range 280 in the reference image portion 252a. The match score for the guide pixel peak position 298 can be the disparity value for the center pixel 270 in the guide node 278a-278g.

[0143] The graph 256 may include a Y-axis 302 and an X-axis 304. The Y-axis 302 may represent match score values ​​determined by the disparity engine 190c for the surrounding pixel blocks 276 at various locations in the local search range 284. The X-axis 304 may represent horizontal (e.g., X) pixel locations of the reference image 252a and the target image 252b for the local search range 284. For example, the X-axis 304 may represent where the match scores are calculated in the stereo pair 252a-252b of images.

[0144] A curve 306 is shown. Curve 306 may represent match score values ​​generated by disparity engine 190c along local search range 284. Lines 308a-308b are shown extending downward from curve 306 to the edge of local search range 284. Curve 306 is shown as generally having the same width as local search range 284 (e.g., a smaller width than curve 296 corresponding to full search range 280). Because disparity engine 190c selects the location of local search range 284 based on guide pixel peak location 282, the match score of surrounding pixel block 276 may be generally higher than the entire local search range 284. Curve 306 is shown as generally having high match score values.

[0145] Points 310a-310c are shown on curve 306. Points 310a-310c may represent matching score values ​​calculated by disparity engine 190c at positions in local search range 284. Since disparity engine 190c may only calculate three positions for surrounding pixel blocks 276 in local search range 284, curve 306 is only shown with calculated matching scores 310a-310c.

[0146] The curve 306 may be a parabola. Although the graph 256 shows the parabola 306 as linear, the disparity engine 190c may only calculate the discrete points 310a-310c. The curve near the peak position of the surrounding pixels 274 may be used as a parabola, as defined by the following equation (eg, EQ1):

[0147] y=ax2+bx+c(EQ1)

[0148] The parallax engine 190c can be configured to draw a parabola 306 based on the calculated matching scores 310a-310c. The calculated matching scores 310a-310c generated according to the three interval block matching operations performed in the local search range can be used as three points defining the parabola 306. For example, the calculated matching scores 310a-310c can be points on the graph 256 (e.g., P1 (x1, y1), P2 (x2, y2), and P3 (x3, y3)). The parallax engine 910c can calculate the coefficient values ​​of a, b, and c that define the parabola 306.

[0149] Point 312 is shown on parabola 306. Point 312 can be a shift offset peak position. Shift offset peak position 312 can be a peak position defined by parabola 306. Disparity engine 190c can be configured to calculate shift offset peak position 312 based on parabola 306 defined by calculated matching scores 310a-310c. For example, shift offset peak position 312 can be determined based on coefficients of parabola 306. The formula for determining shift offset peak position 312 based on parabola 306 can be defined by the following formula (e.g., EQ2). Shift offset peak position 312 can be the disparity of surrounding pixels 274 in guide nodes 278a-278g.

[0150] x=-b / (2*a) (EQ2)

[0151] The parallax engine 190c can be configured to determine the shift offset peak position 312 based on the matching scores 310a-310c using equation EQ2. Since the shift offset peak position 312 is determined according to equation EQ2 (as opposed to a direct block matching operation), the shift offset matching position 312 can be a sub-pixel value. Determining the shift offset peak position 312 at a sub-pixel value can enable accurate values ​​to be generated for the surrounding pixels in the guiding nodes 278a-278g. Accurate values ​​can enable the generation of a stereoscopic image without waveform distortion.

[0152] refer to Figure 6 , a diagram illustrating pixels in a guide node is shown. Guide node 278 is shown. Guide node 278 may be a representative example of a guide node selected by disparity engine 190c in reference image 200a.

[0153] The guide node 278 may include a central pixel 270, a central pixel block 272, and a plurality of surrounding pixels 274a-274x. In the example shown, the guide node 278 may include 25 pixels. For example, the guide node 278 may include a 5×5 pixel grid from the reference image 200a. In the example shown, the central pixel 270 may be a pixel located at the (3, 3) position in the guide node 278. In some embodiments, the guide node 278 may be larger or smaller than the 5×5 pixel data grid. For example, the guide node 278 may be implemented as a 10×10 pixel grid. In the example shown, the guide node 278 may have equal width and height. In some embodiments, the guide node 278 may have a height that is longer than the width or a width that is longer than the height. In some embodiments, the central pixel 270 may not be a single pixel located at the exact center of the guide node 278. In some embodiments, the guide node 278 may have a non-rectangular shape (e.g., a shape that generally forms a diamond, a circle, an ellipse, a star, etc.). The size, shape, and / or number of surrounding pixels in a guidance node 278 may vary according to the design criteria of a particular implementation.

[0154] A central pixel block 272 is shown. The central pixel block 272 may include a central pixel 270 and a plurality of surrounding pixels 274a-274x. In the example shown, the central pixel block 272 may be a 3×3 pixel block within the steering node 278. In the example shown, the central pixel block 272 may include directly adjacent surrounding pixels 274g, 274h, 274i, 274l, 274m, 274p, 274q, and 274r. In some embodiments, the central pixel block 272 may include other pixels (e.g., pixels that are more than one pixel away from the central pixel 270) in the surrounding pixels 274a-274x. The size, shape, and / or number of pixels in the central pixel block 272 may vary according to the design criteria of a particular implementation.

[0155] The disparity engine 190c may be configured to determine the guide pixel peak position 282 using the pixels in the center pixel block 272. After the guide pixel peak position 282 has been determined, the disparity engine 190c may select the local search range 284 and determine the shift offset peak position 312 based on the calculated parabola 306. The shift offset peak position 312 may be determined for each of the surrounding pixels 274a-274x within the guide node 278. For example, for each guide node 278 generated, the disparity engine 190c may perform block matching on the full search range 280 for the center pixel 270, and then perform block matching on the local search range 284 for each of the surrounding pixels 274a-274x (e.g., three calculations). For the example of the 5×5 guide node 278 shown, the disparity engine 190c may perform 72 block matching operations in the local range 284 for the surrounding pixels 274a-274x (e.g., three calculations for each of the 24 surrounding pixels).

[0156] refer to Figure 7 , a diagram illustrating the full search range and the local search range is shown. A full search range 280 is shown. A center pixel block 272 is shown within the full search range 280. A guide pixel peak position 282 in the full search range 280 is shown.

[0157] The disparity engine 190c may perform a block matching calculation for each of a plurality of positions in the full search range. A matching score may be generated based on a comparison of pixels in the central pixel block 272 in the reference image 200a with pixels at X and Y positions in the target image 200b corresponding to each of the positions in the full search range 280. The position with the highest matching score in the full search range 280 may be the guide pixel peak position 282. The number of block matching operations performed for the central pixel block 272 in the full search range 280 may depend on the size of the full search range (e.g., the value of R).

[0158] The surrounding pixel block 276a is shown. The surrounding pixel block 276a is shown adjacent to the central pixel block 272. In one example, the surrounding pixel block 276a may include a 3×5 pixel grid above the central pixel block 272. Figure 6 In the associated illustrated example, the surrounding pixel block 276a for the surrounding pixel 274h (e.g., the pixel directly above the center pixel 270) may include surrounding pixels 274a, 274b, 274c, 274d, 274e, 274f, 274g, 274h, 274i, 274j, 274k, 274l, 274m, 274n and the center pixel 270.

[0159] A local range 284a is shown. The local range 284a may be located at a position near the guide pixel peak position 282. Since the shift offset for the surrounding pixels 274a-274x in the guide node 278 may be generally close to the shift offset for the center pixel 270, the position of the local range 284a may be selected by the disparity engine 190c to be at a position near the guide pixel peak position 282. In the example shown, since the surrounding pixel block 276a is located above the center pixel block 272, the local search range 284a for the surrounding pixel block 276 may be located above the guide node peak position 282.

[0160] The local search range 284a is shown as including block matching calculation positions 350a-350c. The local search range 284a is shown as having a width (e.g., LRW_A). The width LRW_A may include a plurality of pixels (e.g., a range of pixels on the X-axis of the reference image 200a). The size of the block matching calculation positions 350a-350c may be selected based on the width LRW_A.

[0161] The local search range 284a may include three of the block match calculation locations 350a-350c to generate locations for the points 310a-310c on the parabola 306. The local search range 284a may not need to include more than three block match calculation locations 350a-350c because a minimum number of three points 310a-310c may be used to define the parabola 306. A match score may be generated based on a comparison of pixels in the surrounding pixel block 276a in the reference image 200a with pixels at X and Y locations in the target image 200b that correspond to each of the locations of the block match calculation locations 350a-350c.

[0162] The surrounding pixel block 276b is shown. The surrounding pixel block 276b is shown as being diagonally adjacent to the central pixel block 272. In one example, the surrounding pixel block 276b may include a 3×3 pixel grid to the left and below the central pixel block 272. Figure 6 In the example shown in association, the surrounding pixel block 276b for the surrounding pixel 274p (e.g., the pixels directly below and to the left of the central pixel 270) may include surrounding pixels 274k, 274l, 274o, 274p, 274q, 274t, 274u, 274v and the central pixel 270.

[0163] A local range 284b is shown. The local range 284b may be located at a position near the guide pixel peak position 282. Since the shift offset for the surrounding pixels 274a-274x in the guide node 278 may be generally close to the shift offset for the center pixel 270, the position of the local range 284b may be selected by the disparity engine 190c to be at a position near the guide pixel peak position 282. In the example shown, since the surrounding pixel block 276b is located below and to the left of the center pixel block 272, the local search range 284b for the surrounding pixel block 276b may be located below and to the left of the guide node peak position 282.

[0164] The local search range 284b is shown as including block matching calculation positions 350d-350f. The local search range 284b is shown as having a width (e.g., LRW_B). The width LRW_B may include a plurality of pixels (e.g., a range of pixels on the X-axis of the reference image 200a). The size of the block matching calculation positions 350d-350f may be selected based on the width LRW_B.

[0165] The local search range 284b may include three of the block match calculation locations 350d-350f to generate locations for the points 310a-310c on the parabola 306. The local search range 284b may not need to include more than three block match calculation locations 350d-350f because a minimum number of three points 310a-310c may be used to define the parabola 306. A match score may be generated based on a comparison of pixels in the surrounding pixel block 276b in the reference image 200a with pixels at X and Y locations in the target image 200b that correspond to each of the locations of the block match calculation locations 350d-350f.

[0166] The width LRW_A of the local search range 284a is shown as being wider than the width LRW_B of the local search range 284b. In the described example, the size of the surrounding pixel block 276a (e.g., a 3×5 block) can be wider than the size of the surrounding pixel block 276b (e.g., a 3×3 block). The disparity engine 190c can select the size of the local search range 284 based on the size of the surrounding pixel block 276. In some embodiments, each of the local search ranges 284a-284x selected by the disparity engine 190c can have the same size for each of the surrounding pixels 274a-274x. In some embodiments, one or more of the local search ranges 284a-284x selected by the disparity engine 190c can have different sizes for the surrounding pixels 274a-274x. The size (e.g., width) of the local search range 284 and / or the criteria for selecting a specific width of the local search range 284 can vary according to the design criteria of a specific implementation.

[0167] The local search range 284a and / or the local search range 284b are each shown to be near the central pixel peak position 282. The parallax engine 190c can be configured to define a local search range 284 for the surrounding pixels 274a-274x, which generally aligns the center of the local search range 284 with the central pixel peak position 282. The boundary of the local search range 284 can then be extended to the left and / or right of the central pixel peak position 282 by a plurality of pixels. The number of pixels on the left and / or right side of the local search range 284 and / or at this position can be determined in response to the relative position of a particular surrounding pixel 274a-274x relative to the central pixel 270. In some embodiments, the same local search range 284 can be used for each of the surrounding pixels 274a-274x. In some embodiments, more than one local search range position can be used for the surrounding pixels 274a-274x. In one example, if the parabola 296 around the peak location 298 of the center pixel 270 is narrow (e.g., the parabola 296 looks like a sharp pulse), a local search range 284 with a narrower width for the surrounding pixels 274a-274x may be selected. The location and / or width of the local search range 284 and / or the criteria for selecting the location of the local search range 284 may vary depending on the design criteria of a particular implementation.

[0168] In the illustrated example, since two of the surrounding pixel blocks 276a-276b are shown as representative examples, two corresponding local search ranges 284a-284b are shown. In general, the parallax engine 190c can repeat the selection of the position of the local search range 284 near the guide pixel peak position 282 for each surrounding pixel block of each of the surrounding pixels 274a-274x. For example, each local search range 284 for the corresponding surrounding pixels 274a-274x can have a slightly different position (or the same position) near the guide pixel peak position 282 based on the direction and / or distance of the surrounding pixels 274a-274x relative to the center pixel 270.

[0169] In the illustrated example, the full search range 280 and the local search ranges 284a-284b are shown in a horizontal configuration. For example, the full search range 280 and the local search ranges 284a-284b can be used to determine the horizontal shift of pixels between the reference image 200a and the target image 200b. The horizontal shift can be applied to Figure 2 In an embodiment with a vertically oriented stereo camera 100 (such as with Figure 1200a and the target image 200b), the full search range 280 and the local search range 284a-284b may have a vertical orientation for determining a vertical offset (e.g., vertical disparity) between the top image and the bottom image for the reference image 200a and the target image 200b. The orientation of the full search range 280 and the local search range 284 and / or the type of disparity calculated by the disparity engine 190c may vary depending on the orientation of the lenses 106a-106b.

[0170] The disparity engine 190c may select a location for the guide node 278 from pixels in the reference image 200a. The disparity engine 190c may perform a block matching calculation between a central pixel block 272 in the reference image 200a and corresponding pixels (e.g., the same X and Y pixel locations) in the target image 200b. The block matching calculation for the central pixel block 272 may be performed over the entire full search range 280 to determine a guide node pixel peak location 282.

[0171] For example, for each guiding node 278, the disparity engine 190c may clip the central pixel block 272 from the target image 200a. Within the full search range 282, the central pixel block 272 may be shifted pixel by pixel to calculate a matching score with a corresponding pixel in the target image 200b (e.g., an overlapping block in the target image 200b). A matching score parabola 296 (corresponding to Figure 5 270 ) to determine the matching similarity between the reference image 200a and the target image 200b for the central pixel block 272. The peak position in the matching score curve 296 can be used to determine the shift offset disparity for the central pixel 270.

[0172] Since the disparity of the surrounding pixels 274a-274x in the same guide node 278 may be close to the disparity of the center pixel 270, the local search range 284 may be used for each of the surrounding pixels 274a-274x. For each of the surrounding pixels 274a-274x, the disparity engine 190c may perform the following operations: select a local search range 284 near the guide pixel peak position 282; determine a block match score for each of the block match calculation positions 350a-350c in the local search range 284a; and define a parabola 306 using the match scores from each of the block match calculation positions 350a-350c (or the match positions 350d-350f for the local search range 284b or other match positions based on the local search range). The shift offset peak position 312 may be determined by calculating the peak of the parabola 306 defined by the block match calculation positions 350a-350c. The shift offset disparity for the center pixel 270 may be used as a disparity value in a disparity map generated by the disparity engine 190c. The value of the shift offset peak position 312 for each of the surrounding pixels 274a-274x may be used as a disparity value in a disparity map generated by the disparity engine 190c, respectively.

[0173] Block matching techniques may be used to determine disparity values. Block matching techniques may be brute force and have high computational complexity. Block matching techniques may calculate a block matching score over a full search range 280 for each pixel in reference image 200a. For example, if reference image 200a and target image 200b have a size of H, W (e.g., X, Y) and full search range 280 is R, the computational complexity of the block matching technique may be O(H*W*R) to construct an overall disparity map.

[0174] Compared to block matching techniques, the disparity engine 190c can use guide nodes 278 to reduce computational complexity. Block matching calculations can be performed only for the center pixel 270 of each guide node 278 in the reference image 200a in the full search range. Instead of performing block matching calculations for each of the surrounding pixels 274a-274x in each guide node 278 over the full search range 280, the amount of calculations can be reduced by performing three calculations only in the local search range 284. The guide nodes 278 can be defined to ensure that the surrounding pixels 274a-274x have disparity values ​​that are close to the disparity value of the center pixel 270 (e.g., if a random pixel is selected, it may not have a close disparity value).

[0175] For example, if the guide node 278 includes a 10 by 10 pixel grid (e.g., 100 pixels) for an image having a size of H by W and a full search range 280 of R, the disparity engine 190c may determine the disparity value for the center pixel 270 using H×W×R / 100 block matching calculations, and then determine the disparity values ​​for the surrounding pixels using H×W×3×99 / 100 (e.g., approximately 3×H×W) block matching calculations. The total computational complexity for generating disparity values ​​using the guide node may be O(H*W*(0.01R+3)). In an example where the R value for the full search range 280 is 128, the computational complexity of using the guide node 278 may be O(H*W*4.28), which is significantly reduced from the computational complexity of the block matching technique (which may be O(H*W*128)).

[0176] refer to Figure 8 , a diagram illustrating a guide node on a reference video frame is shown. Example 380 is shown. Example 380 may include reference video frame 200a. A plurality of center pixels 270a-270n are shown on reference video frame 200a. The number of center pixels 270a-270n in reference video frame 200a may vary according to design criteria for a particular implementation.

[0177] Center pixels 270a-270n are shown as being spaced apart throughout the video frame 200a. In example 380, center pixel 270a may be at equal pixel distances from adjacent center pixel 270b (e.g., in the X direction) and adjacent center pixel 270h (e.g., in the Y direction). In example 380, center pixel 270f is at different distances from adjacent pixel 270g and adjacent pixel 270e. The distances between center pixels 270a-270n may vary according to design criteria for a particular implementation.

[0178] Dashed boxes 382a-382n are shown on the reference video frame 200a. Dashed boxes 382a-382n may represent guide node positions selected by the disparity engine 190c for the reference video frame 200a. Guide node positions 382a-382n may correspond to corresponding center pixels 270a-270n. For example, guide node position 382a may include center pixel 270a and surrounding pixels near center pixel 270a (e.g., Figure 6 In association with the surrounding pixels 274a-274x shown, the guide node position 382b may include the center pixel 270b and the surrounding pixels near the center pixel 270b, etc.

[0179] The guide node locations 382a-382n are shown covering the entirety of the reference video frame 200a (eg, in FIG. 1 for purposes of clarity and illustration). Figure 8Only guide node positions 382a-382j and 382n are shown in the figure. For example, all pixels of reference video frame 200a can be within guide node positions 382a-382n. Guide node positions 382a-382n are shown to be located at different positions on reference video frame 200a. For example, guide node positions 382a-382n do not overlap with another one of guide node positions 382a-382n. In an example, each of the pixels of reference video frame 200a can be only within one of guide node positions 382a-382n.

[0180] Example 380 can show each of the guide node positions 382a-382n to illustrate the layout and / or position of the guide node positions 382a-382n. In one example, the parallax engine 190c can be configured to operate on the pixels of one of the guide node positions 382a-382n at a time. In some embodiments, the parallax engine 190c can be configured to perform various operations in parallel for more than one of the guide node positions 382a-382n. For example, the parallax engine 270a can perform parallax operations for the surrounding pixels in the guide node position 382a, while performing parallax operations for the center pixel 270b in the guide node position 382b. The order of the parallax operations performed for one or more of the guide node positions 382a-382n can vary according to the design criteria of the specific implementation.

[0181] Arrows 384a-384j are shown. Arrows 384a-384j may illustrate movement of the location of the disparity calculation. In an example, the disparity engine 190c may start the disparity operation from the guide node position 382a (e.g., the upper left corner of the reference video frame 200a). The disparity engine 190c may clip the pixels within the guide node position 382a to perform the disparity operation (e.g., the pixels in the guide node position 382 may be the guide node 278). For example, the pixels of the guide node position 382a may be loaded into the memory 160. The disparity engine 190c may use the full search range 280 to determine the disparity value for the center pixel 270a, and then use the local search range 284 for the disparity values ​​of the surrounding pixels 270a-270x of the guide node position 382a.

[0182] After having performed the disparity operation for the guide node position 382a, the disparity engine 190c may move the guide node 278 to the guide node position 382b (e.g., as indicated by arrow 384a). The disparity engine 190c may clip the pixels within the guide node position 382b to perform the disparity operation (e.g., the pixels in the guide node position 382b may be the guide node 278). For example, the pixels of the guide node position 382b may be loaded into the memory 160. The disparity engine 190c may use the full search range 280 to determine the disparity value for the center pixel 270b, and then use the local search range 284 for the disparity values ​​of the surrounding pixels 270a-270x of the guide node position 382b.

[0183] After having performed the disparity operation for guide node position 382b, disparity engine 190c may move guide node 278 to guide node position 382c (e.g., as shown by arrow 384b). Disparity engine 190c may perform similar disparity operations for guide node positions 382c-382g. For example, guide node 278 may shift (e.g., slide) from left to right across the top row of guide node positions 382a-382g until the entire width of reference video frame 200a has been calculated. After disparity engine 190c reaches the end of the top row of guide node positions at guide node position 382g, disparity engine 190c may move to the next row of guide node positions 382a-382n in reference video frame 200a.

[0184] Guidance node position 382g is shown as having a smaller width than guidance node position 382a-382f. In some embodiments, each of guidance node position 382a-382n can have the same width (e.g., parallax engine 190c can select the size of guidance node position 382a-382n so that each guidance node position 382a-382n has equal height and width). In some embodiments, most of guidance node position 382a-382n can be selected to be a specific size (e.g., based on the capacity available in memory 160), and if the specific size is not evenly divided, one or more guidance node positions 382a-382n with a smaller size can exist. The size of guidance node position 382a-382n can vary according to the design criteria of a specific implementation.

[0185] Arrows 386a-386b are shown representing the movement of the guide node 278 from the last position in the top row of guide node positions (e.g., 382g) to the first position in the second row of guide node positions (e.g., 382h). Similar parallax operations may be performed by the parallax engine 190c for the next row of guide node positions (e.g., guide node positions 382h-382j are shown for the second row of guide node positions). The parallax engine 190c may shift the guide node 278 across the next row of guide node positions (e.g., the movement of the guide node may be represented by arrows 384h-384j).

[0186] The parallax engine 190c can continue to slide the guide node 278 across each row of the guide node positions 382a-382n of the reference video frame 200a until it reaches the guide node position 382n (e.g., the guide node position in the lower right corner of the reference video frame 200a). When the parallax operation has been completed for the last guide node position 382n, the parallax operation can be completed for the reference video frame 200a. In example 380, the movement of the guide node 278 can be from left to right (or from right to left) across each row of the guide node positions 382a-382n from top to bottom. In some embodiments, the movement of the guide node 278 can be from top to bottom along each column of the guide node positions 382a-382n from left to right (or from right to left). The direction of the movement of the guide node 278 across the guide node positions 382a-382n can vary according to the design criteria of the specific implementation.

[0187] refer to Fig. 9 , a diagram illustrating a disparity map is shown. A disparity map 400 is shown. The disparity map 400 may be an illustrative example of a disparity map generated by the processor 102. The disparity map 400 may show the displacement of pixels between the reference image 200a and the target image 200b. The disparity values ​​may be within the R value of the full search range 280. In an example, if the R value for the full search range 280 is 128, the disparity values ​​for the pixels in the reference image 200a may be values ​​between 0 and 128.

[0188] In the example shown, the disparity map 400 may be generated in response to pixel data captured by the capture devices 104a-104b of the human head 404. The disparity map 400 may include regions 402-412. For purposes of illustration, the disparity map 400 may be a simplified version of a disparity map generated by the processor 102. In some embodiments, the regions 402-412 may be grayscale representations of pixel values ​​in a range of 0 to 255 (e.g., integer values) covering a range of disparities. A guide node may be used to determine the disparity values ​​of the disparity map 400.

[0189] In an example, region 402 may represent pixels having disparity values ​​in a first range of values ​​(e.g., locations behind person 404), region 406 may represent pixels having disparity values ​​in a second range of values ​​(e.g., locations generally corresponding to the top of the hair and / or head of person 404), region 408 may represent pixels having disparity values ​​in a third range of values ​​(e.g., locations generally corresponding to the face of person 404), region 410 may represent pixels having disparity values ​​in a fourth range of values ​​(e.g., locations generally corresponding to the nose of person 404), and region 412 may represent pixels having disparity values ​​in a fifth range of values ​​(e.g., locations generally corresponding to the shoulders of person 404). In general, pixels within corresponding regions 402-412 may generally have similar (but not necessarily matching) disparity values. Because the disparity values ​​in corresponding regions 402-412 may generally have similar values, local search region 284 may be used to guide surrounding pixels 274a-274x in node 278.

[0190] Determining the shift offset peak position 312 for the surrounding pixels 274a-274x based on the calculated parabola 306 can enable the disparity engine 190c to generate disparity values ​​at a sub-pixel level. Generating disparity values ​​at a sub-pixel level can enable the disparity map 400 to be generated with less noise than when the disparity map is generated using block matching techniques. For example, noise can be represented as outliers in the regions 402-412.

[0191] In some embodiments, the difference between the disparity values ​​determined using block matching techniques and using a guide node may be small. In one example, the disparity value determined using a block matching technique may be approximately 80, while the same disparity value determined using a guide node may be 80.3. The small difference may be due to the sub-pixel accuracy achieved by the guide node operation. Although sub-pixel accuracy may not be visible in the disparity map 400 (e.g., due to a small difference in value), sub-pixel accuracy may have an impact on the generated stereoscopic image. For example, sub-pixel accuracy can prevent waveform distortion in a stereoscopic image.

[0192] refer to Fig.10 , a diagram illustrating a stereoscopic image is shown. An example stereoscopic image 450 is shown. The stereoscopic image 450 may include a surface of a 3D object. The stereoscopic image 450 may be composed of points 452a-452n. The points 452a-452n may represent the surface of the 3D object. In the example shown, the 3D object may include a face 460. A nose 462 is shown on the face 460.

[0193] Points 452a-452n may be generated by the disparity engine 190c in response to disparity values ​​generated using the guidance nodes. The disparity values ​​may be used by the disparity engine 190c along with other values ​​(e.g., focal lengths of the capture devices 104a-104b, distances between the two capture devices 104a-104b, etc.) to represent the surface of the 3D object. Generating disparity values ​​using guidance nodes may ensure that mismatches (e.g., finding non-optimal disparity values) are limited.

[0194] A dashed box 470 is shown. The dashed box 470 may include a subset of points 452a-452n corresponding to the cheeks and nose 462 of the face 460. The dashed box 470 may represent a complex area of ​​the face 460 (e.g., a relatively large change in distance from the lenses 106a-106b in a small area of ​​the face 460). The points 452a-452n in the box 470 show a relatively smooth transition. Mismatches may result in noise (e.g., due to outliers). Using the guide node 278 to calculate the disparity values ​​may ensure that there are few generated mismatched disparity values ​​(e.g., fewer than when using block matching techniques). There may be no waveform distortion in the box 470.

[0195] Typically, the risk of a mismatch may be lower when using the guide node 278 to generate disparity values. If a mismatch does occur, the processor 102 may be configured to detect the mismatch. The mismatch may be corrected using post-processing implemented by the processor 102. In one example, a full range search may be performed to detect a mismatch for surrounding pixels 274a-274x.

[0196] When using the guide node, the sub-pixel accuracy of the generated disparity values ​​can prevent waveform distortion. The waveform distortion can appear as a "slice" of the points 452a-452n. In box 470, the points 452a-452n are shown as being evenly distributed (e.g., smoothly transitioned). In an example, waveform distortion may exist in a stereo image generated using a block matching technique, but may not exist in the stereo image 450 shown.

[0197] refer to Fig.11 , showing a method (or process) 500. The method 500 can construct a disparity map using a guide node. The method 500 generally includes a step (or state) 502, a step (or state) 504, a step (or state) 506, a step (or state) 508, a step (or state) 510, a step (or state) 512, a step (or state) 514, a step (or state) 516, a step (or state) 518, a decision step (or state) 520, a decision step (or state) 522, a step (or state) 524, and a step (or state) 526.

[0198] Step 502 may start method 500. In step 504, processor 102 may receive pixel data PIXELD_A-PIXELD_B from a stereo pair 104a-104b of a capture device. Next, in step 506, processor 102 may generate reference image 200a and target image 200b. For example, a video processing pipeline of processor 102 may be configured to perform video processing operations on pixel data PIXELD_A-PIXELD_B to generate reference image 200a and target image 200b. Next, method 500 may move to step 508.

[0199] In step 508, disparity engine 190c may select a next guide node 278 on reference image 200a. In an example, disparity engine 190c may select a pixel from one of guide node positions 382a-382n as the next position of guide node 278. Guide node 278 may correspond to the position of center pixel 270. Next, in step 510, disparity engine 190c may use full search range 280 to determine guide node pixel peak position 298 for center pixel 270 of guide node 278. Next, method 500 may move to step 512.

[0200] In step 512, the disparity engine 190c may select the next of the surrounding pixels 274a-274x within the guiding node 278. Next, in step 514, the disparity engine 190c may select a local search range 284 (e.g., a size and / or position of the local search range 284) for the selected one of the surrounding pixels 274a-274n. In step 516, the disparity engine 190c may perform a block matching operation in the local search range 284 corresponding to the selected one of the surrounding pixels 274a-274x. Next, in step 518, the disparity engine 190c may generate a disparity value for the selected one of the surrounding pixels 274a-274x in the current guiding node 178. Next, the method 500 may move to a decision step 520.

[0201] In decision step 520, disparity engine 190c may determine whether there are more surrounding pixels 274a-274x within the current guiding node 278. If there are more surrounding pixels 274a-274x within the current guiding node 278, the method may return to step 512. For example, disparity engine 190c may determine a disparity value for each of the surrounding pixels 274a-274x within the guiding node 278. If there are no more surrounding pixels 274a-274x (e.g., disparity values ​​have been generated for all surrounding pixels 274a-274x within the guiding node 278), the method 500 may move to decision step 522.

[0202] In decision step 522, disparity engine 190c may determine whether there are more guide node positions 382a-382n. If there are more guide node positions 382a-382n, method 500 may return to step 508. For example, disparity engine 190c may determine disparity values ​​for pixels within each of guide node positions 382a-382n until disparity values ​​have been generated for all pixels in reference video frame 200a. If there are no more guide node positions 382a-382n (e.g., disparity values ​​have been generated for all pixels), method 500 may move to step 524. In step 524, disparity engine 190c may output disparity map 400. Next, method 500 may move to step 526. Step 526 may end method 500.

[0203] refer to Fig.12 , a method (or process) 550 is shown. The method 550 can calculate the disparity value for the surrounding pixels. The method 550 generally includes a step (or state) 552, a step (or state) 554, a step (or state) 556, a step (or state) 558, a step (or state) 560, a decision step (or state) 562, a step (or state) 564, a step (or state) 566, a step (or state) 568, a decision step (or state) 570, and a step (or state) 572.

[0204] Step 552 may start method 550. In step 554, disparity engine 190c may determine center pixel peak position 298 for guide node 278. Next, method 550 may move to step 556.

[0205] In step 556, the disparity engine 190c may select the next of the surrounding pixels 274a-274x within the guiding node 278. Next, in step 558, the disparity engine 190c may select a local search range 284 near the center pixel peak position 298 for the selected one of the surrounding pixels 274a-274x (e.g., near the same X value for a horizontal stereo camera or near the same Y value for a vertical stereo camera). In step 560, the disparity engine 190c may perform a block matching operation between the surrounding pixel block 276 and one of the positions 350a-350c of the local search range 284. Next, the method 550 may move to a decision step 562.

[0206] In decision step 562, disparity engine 190c may determine whether three block matching operations have been performed for surrounding pixel block 276. For example, disparity engine 190c may generate a matching score for each of three positions 350a-350c. If three block matching operations have not been performed, method 550 may return to step 560 and determine a matching score for the next position in positions 350a-350c of local search range 284. If all three block matching operations have been performed, method 550 may move to step 564.

[0207] In step 564, the disparity engine 190c may use the block matching scores 310a-310c generated from the block matching operations performed at the three locations 350a-350c to determine coefficients for defining the parabola 306. Next, in step 566, the disparity engine 190c may calculate (e.g., based on equation EQ2) the peak 312 of the parabola 306. In step 568, the disparity engine 190c may use the peak 312 of the parabola as a shift offset peak position for the selected one of the surrounding pixels 274a-274x. Next, the method 550 may move to decision step 570.

[0208] In decision step 570, disparity engine 190c may determine whether there are more surrounding pixels 274a-274x in guide node 278. If there are more surrounding pixels 274a-274x (e.g., more disparity values ​​need to be determined), method 550 may return to step 556. If there are no more surrounding pixels 274a-274x (e.g., disparity values ​​have been determined for all pixels in guide node 278), method 550 may move to step 572. Step 572 may end method 500. For example, all pixels in the current guide node 278 may be calculated, and the guide node may be moved to another position of the guide node position 382a-382n.

[0209] refer to Fig.13 , a method (or process) 600 is shown. The method 600 can calculate the disparity value for the center pixel. The method 600 generally includes step (or state) 602, step (or state) 604, step (or state) 606, step (or state) 608, step (or state) 610, decision step (or state) 612, step (or state) 614, step (or state) 616, step (or state) 618 and step (or state) 620.

[0210] Step 602 may begin method 600. In step 604, disparity engine 190c may select a center pixel 270 for a guide node 278. Next, in step 606, disparity engine 190c may select a pixel block (e.g., center pixel block 272) surrounding center pixel 270. For example, center pixel block 272 may include center pixel 270 and a subset of surrounding pixels 274a-274x. Next, method 600 may move to step 608.

[0211] In step 608, the disparity engine 190c may shift the central pixel block 272 to the next position in the full search range 280. For example, the disparity engine may start from the central pixel block 272 at the leftmost position (or the rightmost position) in the full search range 280, and shift the central pixel block 272 from left to right (or from right to left) across the entire full search range 280. Next, in step 610, the disparity engine 190c may calculate a matching score for the central pixel block 272 at the position in the reference image 200a and the corresponding position in the target image 200b. Next, the method 600 may move to a decision step 612.

[0212] In decision step 612, disparity engine 190c may determine whether there is another position in full search range 280. If there are more positions in full search range 280, method 600 may return to step 608. For example, disparity engine 190c may determine a match score for each position in full search range 280. If there are no more positions in full search range 280 (e.g., a match score has been generated for each position so that curve 296 can be generated), method 600 may move to step 614.

[0213] In step 614, the disparity engine 190c may determine the highest match score generated over the full search range 280. The highest match score may be the highest value of the curve 296. Next, in step 616, the disparity engine 190c may use the highest match score as the guide node peak position 298. In step 618, the disparity engine 190c may use the guide node peak position 298 as the disparity value for the center pixel 270 of the guide node 278. Next, the method 600 may move to step 620. Step 620 may end the method 600. For example, after the guide node pixel peak position 298 has been determined, the disparity module 190c may calculate disparity values ​​for the surrounding pixels 274a-274x, such as with Fig.12 Described in relation to each other.

[0214] Depend on Figure 1-13 The functions performed by the diagrams may be implemented using one or more of a conventional general purpose processor, a digital computer, a microprocessor, a microcontroller, a RISC (Reduced Instruction Set Computer) processor, a CISC (Complex Instruction Set Computer) processor, a SIMD (Single Instruction Multiple Data) processor, a signal processor, a central processing unit (CPU), an arithmetic logic unit (ALU), a video digital signal processor (VDSP), and / or similar computing machines programmed according to the teachings of the specification, as will be apparent to those skilled in the relevant art(s). A skilled programmer can readily prepare appropriate software, firmware, coding, routines, instructions, opcodes, microcodes, and / or program modules based on the teachings of the present disclosure, as will be apparent to those skilled in the relevant art(s). The software is typically executed from one or more media by one or more of the processors of the machine implementation.

[0215] The present invention may also be implemented by preparing an ASIC (application specific integrated circuit), a platform ASIC, an FPGA (field programmable gate array), a PLD (programmable logic device), a CPLD (complex programmable logic device), a sea-of-gate, an RFIC (radio frequency integrated circuit), an ASSP (application specific standard product), one or more monolithic integrated circuits, one or more chips or dies arranged as a flip-chip module and / or a multi-chip module, or by interconnecting an appropriate network of conventional component circuits, as described herein, modifications of which will be apparent to those skilled in the art.

[0216] Thus, the present invention may also include a computer product, which may be one or more storage media and / or one or more transmission media, which may include instructions that may be used to program a machine to perform one or more processes or methods according to the present invention. The execution of the instructions contained in the computer product by the machine and the operation of the surrounding circuitry may convert input data into one or more files on the storage medium and / or one or more output signals representing a physical object or substance, such as audio and / or visual depictions. The storage medium may include, but is not limited to, any type of disk, including floppy disks, hard drives, magnetic disks, optical disks, CD-ROMs, DVDs, and magneto-optical disks, as well as circuits such as ROM (read-only memory), RAM (random access memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), UVPROM (ultraviolet erasable programmable ROM), flash memory, magnetic cards, optical cards, and / or any type of medium suitable for storing electronic instructions.

[0217] The element of the present invention can form a part or all of one or more equipment, units, components, systems, machines and / or devices. Equipment can include but is not limited to servers, workstations, storage array controllers, storage systems, personal computers, laptop computers, notebook computers, palmtop computers, cloud servers, personal digital assistants, portable electronic devices, battery-powered equipment, set-top boxes, encoders, decoders, code converters, compressors, decompressors, preprocessors, postprocessors, transmitters, receivers, transceivers, cryptographic circuits, cellular phones, digital cameras, positioning and / or navigation systems, medical equipment, heads-up displays, wireless devices, audio recordings, audio storage and / or audio playback equipment, video recordings, video storage and / or video playback equipment, gaming platforms, peripherals and / or multi-chip modules. (Multiple) technicians in the relevant field will understand that the element of the present invention can be implemented in other types of equipment to meet the standards of specific applications.

[0218] When used in conjunction with "is" and a verb herein, the terms "may" and "generally" are intended to convey the intention that the description is exemplary and is considered broad enough to cover both the specific examples presented in the present disclosure and alternative examples that can be derived based on the present disclosure. The terms "may" and "generally" as used herein should not be interpreted as necessarily implying the desirability or possibility of omitting the corresponding elements.

[0219] While the invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the invention.

Claims

1. A device for constructing a disparity map, comprising: an interface configured to receive pixel data; as well as A processor configured to: (i) generate a reference image and a target image based on the pixel data; (ii) perform a parallax operation on the reference image and the target image; and (iii) constructing a disparity map in response to the disparity operation, wherein the disparity operation comprises: (a) selecting a guide node from the pixel data, the guide node comprising a guide node pixel and a plurality of surrounding pixels; (b) determining a guide node pixel peak position for the guide node pixel by performing a full range search in the reference image and the target image; (c) calculating a shift offset peak position for each of the surrounding pixels in the guide node by performing a block matching operation in a local range near the guide node pixel peak position; and (d) generating a value in the disparity map for the pixel data in the guide node in response to (i) the guide node pixel peak position for the guide node pixel and (ii) the shift offset peak position for each of the surrounding pixels.

2. The device according to claim 1, wherein: The shift offset peak position for the surrounding pixels is determined in response to the following operations: (i) selecting the local range near the peak position of the guide node pixel; (ii) generating coefficients in response to performing a block matching operation at three positions in the local range by comparing blocks of the pixel data near one of the surrounding pixels in the reference image and the target image; (iii) calculating the shift offset peak position in response to the coefficients; and (iv) repeating (i)-(iii) for each of the surrounding pixels.

3. The device according to claim 2, wherein: (i) the coefficients define a parabola; and (ii) the shift offset peak location is determined in response to calculating a peak location of the parabola.

4. The device according to claim 1, wherein: The full-range search includes: (i) shifting the position of the pixel block including the guide node pixel across the search range; (ii) calculating one of multiple matching scores for one of multiple positions of the pixel block including the guide node pixel in the reference image and the corresponding position in the target image; (iii) repeating (i)-(ii) for each of the multiple positions in the search range; and (iv) selecting the highest value from the matching scores as the peak position of the guide node pixel.

5. The device according to claim 4, wherein: The local range is smaller than the search range.

6. The device according to claim 1, wherein: The disparity operation is configured to construct the disparity map with sub-pixel accuracy.

7. The device according to claim 1, wherein: The disparity operation includes post-processing to prevent waveform distortion in the disparity map.

8. The device according to claim 7, wherein: The post-processing includes implementing a parabola to calculate the matching position.

9. The device according to claim 1, wherein: The disparity operation is configured to generate the disparity map at least 20 times faster than using the block matching operation on a full video frame.

10. A method for determining a disparity value from a stereoscopic image, comprising the following steps: Receiving pixel data at the interface; Using a processor, generating a reference image and a target image based on the pixel data; Using the processor, performing a parallax operation on the reference image and the target image; as well as Using the processor, a disparity map is constructed in response to the disparity operation, wherein the disparity operation includes: (a) selecting a guide node from the pixel data, the guide node including a guide node pixel and a plurality of surrounding pixels; (b) determining a guide node pixel peak position for the guide node pixel by performing a full range search in the reference image and the target image; (c) calculating a shift offset peak position for each of the surrounding pixels in the guide node by performing a block matching operation in a local range near the guide node pixel peak position; and (d) generating a value in the disparity map for the pixel data in the guide node in response to (i) the guide node pixel peak position for the guide node pixel and (ii) the shift offset peak position for each of the surrounding pixels.

11. The method according to claim 10, wherein: The shift offset peak position for the surrounding pixels is determined in response to the processor performing the following steps: (i) selecting the local range near the peak position of the guiding node pixel, (ii) generating coefficients in response to performing a block matching operation at three locations in the local range by comparing blocks of the pixel data in the vicinity of one of the surrounding pixels in the reference image and the target image, (iii) calculating the shift offset peak position in response to the coefficient, and (iv) Repeat (i)-(iii) for each of the surrounding pixels.

12. The method according to claim 11, wherein: (i) the coefficients define a parabola; and (ii) the shift offset peak location is determined in response to calculating a peak location of the parabola.

13. The method according to claim 10, wherein: The full range search includes the processor performing the following steps: (i) shifting the position of the pixel block including the guiding node pixel across the search range, (ii) calculating one of a plurality of matching scores for one of a plurality of positions of the pixel block including the guiding node pixel in the reference image and a corresponding position in the target image, (iii) repeating (i)-(ii) for each of the plurality of positions in the search range, and (iv) Selecting the highest value from the matching scores as the guide node pixel peak position.

14. The method according to claim 13, wherein: The local range is smaller than the search range.

15. The method according to claim 10, wherein: The disparity operation is configured to construct the disparity map with sub-pixel accuracy.

16. The method according to claim 10, wherein: The disparity operation includes post-processing to prevent waveform distortion in the disparity map.

17. The method according to claim 16, wherein: The post-processing includes implementing a parabola to calculate the matching position.

18. The method according to claim 10, wherein: The disparity operation is configured to generate the disparity map at least 20 times faster than using the block matching operation on a full video frame.

Images