Unsupervised Multi-Scale Disparity / Optical Flow Fusion

Abstract

A device for implementing unsupervised multi-scale disparity / optical flow fusion includes an interface and a processor. The interface can be configured to receive an image from a capture device. The processor can be configured to: (i) extract features from the image; (ii) generate a fused disparity map in response to the extracted features; (iii) generate a regenerated image by performing a warp on a first subset of the image based on (a) the fused disparity map and (b) a first parameter; (iv) perform a comparison between the sample image and an image in the first subset of the image; (v) perform a classification of the sample image based on a second parameter; and (vi) update the first parameter and the second parameter in response to whether the classification is correct, wherein (a) the classification includes indicating whether the sample image is an image in the second subset of the image or a regenerated image in the regenerated image; and (b) the regenerated image is used as a training data set.

Description

Technical Field

[0001] The present invention relates generally to sensor fusion, and more particularly to methods and / or apparatus for implementing unsupervised multi-scale disparity / optical flow fusion. Background Art

[0002] In dual-camera stereo reconstruction, the accuracy and density of disparity / optical flow maps are crucial for advanced algorithms. When disparity is calculated only in high-resolution images, the calculation may be affected by noise and dense disparity / optical flow maps cannot be obtained. When disparity is calculated only in low-resolution images, a denser but less accurate disparity is obtained. Typically, disparity is calculated in both high-resolution and low-resolution images and the results are merged together. However, it is challenging to manually design a suitable fusion algorithm. Conventional convolutional neural networks (CNNs) require ground truth disparity to train the network. However, such ground truth disparity for training datasets is not easy to collect.

[0003] It is expected to achieve unsupervised multi-scale disparity / optical flow fusion. Summary of the invention

[0004] The present invention covers aspects relating to an apparatus comprising an interface and a processor. The interface may be configured to receive an image from a capture device. The processor may be configured to: (i) extract features from the image; (ii) generate a fused disparity map in response to the extracted features; (iii) generate a regenerated image by performing a warp on a first subset of the image based on (a) the fused disparity map and (b) a first parameter; (iv) perform a comparison between the sample image and an image in the first subset of the image; (v) perform a classification of the sample image based on a second parameter; and (vi) update the first parameter and the second parameter in response to whether the classification is correct, wherein (a) the classification includes indicating whether the sample image is an image in the second subset of the image or a regenerated image in the regenerated image; and (b) the regenerated image is used as a training data set.

[0005] In some embodiments of the apparatus aspects described above, (i) the capture device includes a first camera and a second camera configured as a stereo camera; (ii) a first subset of images may be captured by the first camera; and (iii) a second subset of images may be captured by the second camera. In an embodiment implementing a stereo camera, (i) the first subset of images includes a left image; (ii) the second subset of images includes a right image; and (iii) the training data set includes data for disparity calculation.

[0006] In some embodiments of the apparatus aspects described above, (i) the first subset of images includes images captured at an earlier time in the sequence of images; (ii) the second subset of images includes images captured at a later time in the sequence of images compared to the first subset of images; and (iii) the training data set includes optical flow information.

[0007] In some embodiments of the apparatus aspects described above, the sample image is randomly selected from the second subset of images or the regenerated image. In some embodiments where the sample image is randomly selected, (a) the first parameter is updated to adjust the regenerated image to reduce the probability that the sample image is classified as correct; and (b) the second parameter is updated to increase the probability that the sample image is classified as correct. In some embodiments where the sample image is randomly selected, when the probability is greater than a threshold amount, the regenerated image is used as a training data set. In some embodiments where the regenerated image is used as a training data set when the probability is greater than a threshold amount, the threshold amount can be approximately 50%.

[0008] In some embodiments of the apparatus aspects described above, the processor is configured to implement: (i) a first neural network to implement a first model based on a first parameter; and (ii) a second neural network to implement a second model based on a second parameter. In embodiments implementing the first neural network and the second neural network, (i) the first neural network may be configured to: (a) extract features from a first subset of images and a second subset of images; (b) generate a fused disparity map in response to the extracted features; and (c) generate a regenerated image; and (ii) the second neural network may be configured to perform: (a) a comparison of a sample image with an image in the first subset of images; and (b) a classification of the sample image based on the second parameter. In some embodiments implementing the first neural network and the second neural network, (i) the first neural network typically implements a generative neural network model; and (ii) the second neural network typically implements a discriminative neural network model. In embodiments implementing the generative neural network model and the discriminative neural network model, the generative neural network model and the discriminative neural network model may be configured as an unsupervised generative adversarial network. In some embodiments implementing the generative neural network model and the discriminative neural network model, the processor may be configured to operate using the generative neural network model and the discriminative neural network model in a training mode of operation to generate a training data set. In some embodiments, the processor is further configured to operate in the data generation mode of operation using the generative neural network model and disabling the discriminative neural network model. In some embodiments, the processor operates in the data generation mode of operation after the generative neural network model has been sufficiently trained during the training mode of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the invention will become apparent from the following detailed description, appended claims and accompanying drawings.

[0010] Figure 1 is a diagram illustrating a processor in the context of a stereo camera application according to an example embodiment of the present invention.

[0011] Figure 2 is a diagram illustrating a processor in the context of a temporal image sequence application according to an example embodiment of the present invention.

[0012] Figure 3 is a diagram illustrating a processor in a training configuration according to an example embodiment of the present invention.

[0013] Figure 4 is a diagram showing a process of generating a multi-scale disparity map.

[0014] Figure 5 is a diagram showing a process of generating a fused disparity map.

[0015] Figure 6 FIG. 1 is a diagram showing a process of generating a reconstructed image using a fused disparity map.

[0016] Figure 7 is a diagram illustrating a process according to an example embodiment of the present invention.

[0017] Figure 8 is a diagram illustrating another process according to an example embodiment of the present invention.

[0018] Fig. 9 is a diagram showing the process of generating a multi-scale optical flow map.

[0019] Fig.10 is a diagram showing a process of generating a fused optical flow map.

[0020] Fig.11 FIG. 1 is a diagram showing a process of generating a reconstructed image using a fused optical flow map.

[0021] Fig.12 is a diagram illustrating a process according to an example embodiment of the present invention.

[0022] Fig.13 is a diagram illustrating another process according to an example embodiment of the present invention.

[0023] Fig.14 is a diagram illustrating an example of a computer system in which an unsupervised training process according to an example embodiment of the present invention may be implemented.

[0024] Fig.15is a diagram of a camera system illustrating an example implementation of a computer vision system in which a multi-scale disparity / optical flow fusion algorithm according to an example embodiment of the present invention may be implemented.

[0025] Fig.16 is a diagram illustrating a context in which a disparity / optical flow fusion network according to an example embodiment of the present invention may be implemented.

[0026] Fig.17 It is shown Fig.16 Figure 2 is a diagram of a general implementation of a hardware engine. DETAILED DESCRIPTION

[0027] Embodiments of the present invention include providing unsupervised multi-scale disparity / optical flow fusion that can (i) utilize a novel generative adversarial network (GAN) based algorithm; (ii) create a stereo pair of images from a single image; (iii) predict the motion of objects in the image; (iv) avoid the need for a ground truth dataset; (v) improve the density and accuracy of disparity map and optical flow map computations and / or (vi) be implemented as one or more integrated circuits.

[0028] In various embodiments, a new generative adversarial network (GAN) based algorithm may be implemented to perform unsupervised multi-scale disparity / optical flow fusion. Optical flow and stereo disparity are two basic and related ways to analyze images in computer vision systems. Optical flow measures the apparent motion of a point between two images, while stereo disparity measures the depth of an object relative to two cameras. Typically, optical flow measures motion over time, while stereo disparity measures displacement at a point in time. Optical flow and stereo disparity are commonly used in computer vision tasks, including but not limited to automotive and robotic navigation and video analysis.

[0029] refer to Figure 1 , a diagram illustrating a processor in the context of a stereo camera application according to an example embodiment of the present invention is shown. In an example, circuit 100 may implement a processor according to an example embodiment of the present invention. In an example, circuit 100 may be configured to receive an image from camera assembly 90. In an example, circuit 100 may receive a first signal (e.g., R_IMAGES) and a second signal (e.g., L_IMAGES) from camera assembly 90. Signal R_IMAGES may transmit a first subset of images from camera assembly 90 to circuit 100. Signal L_IMAGES may transmit a second subset of images from camera assembly 90 to circuit 100. In an example in which camera 90 is configured as a stereo camera, signals R_IMAGES and L_IMAGES may include corresponding right and left images captured by camera assembly 90, respectively.

[0030] In an example, the camera assembly 90 may include a first camera 92 and a second camera 94. In an example embodiment, the first camera 92 and the second camera 94 may be configured as a stereo camera pair. For example, the first camera 92 and the second camera 94 may be installed so that the optical axis of the first camera 92 and the optical axis of the second camera 94 are at a predetermined angle to each other.

[0031] In various embodiments, circuit 100 may be configured to calculate a disparity value between corresponding images of signal R_IMAGES and signal L_IMAGES. In an example, the disparity value may be utilized for object detection and / or reconstruction. In another example, the disparity value may be utilized for object tracking, depth measurement, object velocity and / or motion determination, etc. In general, disparity includes depth information that may be utilized in any application requiring three-dimensional (3D) information (e.g., 3D reconstruction).

[0032] In an example embodiment, circuit 100 may include block (or circuit) 102, block (or circuit) 104, block (or circuit) 106, and block (or circuit) 108. Blocks 102 and 104 may be implemented using an artificial neural network (ANN). In an example embodiment, blocks 102 and 104 may be implemented as convolutional neural networks (CNNs) or deep convolutional neural networks (DCNNs). In an example, each of the neural networks of blocks 102 and 104 may be implemented using a directed acyclic graph (DAG) and corresponding weighted sets 102a and 104a. In an example embodiment, the DAG may be executed using a hardware engine of processor 100. In an example, the following may be used in combination with Figure 3 The described unsupervised training process determines the corresponding weighted sets 102a and 104a.

[0033] In an example embodiment, circuit 106 may be implemented as a disparity engine. Circuit 106 may be implemented using any hardware, software, and / or combination of hardware and software that calculates disparity (including but not limited to conventional disparity calculation circuits). In an example, hardware implementations of circuit 106 may include but are not limited to field programmable gate arrays (FPGAs) and application specific integrated circuits (ASICs). In an example embodiment, circuit 108 may be implemented as an object detection / reconstruction circuit. Circuit 108 may be implemented using any hardware, software, and / or combination of hardware and software that detects and / or reconstructs objects within an image (including but not limited to conventional techniques for detecting and / or reconstructing objects). In an example, hardware implementations of circuit 108 may include but are not limited to field programmable gate arrays (FPGAs) and application specific integrated circuits (ASICs).

[0034] In an example embodiment, block 102 may be implemented as a generative neural network model, and block 104 may be implemented as a discriminative neural network model. In various embodiments, block 102 may be utilized during the training and inference (deployment) phases of circuit 100, while block 104 may be utilized only during the training phase. In some embodiments, block 104 may exist, but block 104 is inactive after deployment (e.g., as shown by the dashed line). In some embodiments, block 104 may be omitted from an instance of circuit 100 being deployed. In various embodiments, block 102 may have an input that may receive multiple scaled disparity maps (e.g., a disparity map for each scale of a disparity map pyramid) and an output that may present a fused disparity map. Block 102 may generate a fused disparity map based on multiple scaled disparity maps based on weighting 102a.

[0035] In an example embodiment, circuit 106 may have a first input that may receive signal R_IMAGES, a second input that may receive signal L_IMAGES, and an output that may transmit the plurality of scaled disparity maps to block 102. The output of block 102 may be coupled to an input of circuit 108. In an example embodiment, signals R_IMAGES and L_IMAGES may be transmitted to circuit 106 using one or more memories or buffers (not shown). In another example, signals R_IMAGES and L_IMAGES may also be transmitted to block 102 to further improve the fused disparity values ​​generated by block 102.

[0036] refer to Figure 2 , a diagram illustrating a processor in the context of a temporal image sequence application according to an example embodiment of the present invention is shown. In an example, circuit 100' can implement a processor according to another example embodiment of the present invention. In an example, circuit 100' can be configured to receive an image from camera assembly 96. In an example, circuit 100' can receive a signal (e.g., IMAGES) from camera assembly 96. Signal IMAGES can transmit a sequence of images from camera assembly 96 to circuit 100'.

[0037] In various embodiments, circuit 100' can be configured to calculate optical flow values ​​between pairs of images of signal IMAGES. In an example, the optical flow values ​​can be utilized for object detection and / or reconstruction. In another example, the optical flow values ​​can be utilized for object tracking, depth measurement calculations, object speed and / or motion determination, etc. In general, optical flow includes depth information that can be utilized in any application requiring three-dimensional (3D) information (e.g., 3D reconstruction).

[0038] In an example embodiment, circuit 100' may include block (or circuit) 102, block (or circuit) 104, block (or circuit) 110, and block (or circuit) 112. In an example embodiment, circuit 110 may be implemented as an optical flow engine. Circuit 110 may be implemented using any hardware, software, and / or a combination of hardware and software (including conventional optical flow calculation circuits) for calculating optical flow. In an example, the hardware implementation of circuit 110 may include, but is not limited to, a field programmable gate array (FPGA) and an application specific integrated circuit (ASIC). In an example embodiment, circuit 112 may be implemented as an object detection / reconstruction circuit. In various embodiments, circuit 112 is generally implemented in a different manner from circuit 108. In an example, circuit 108 may be configured to receive a one-dimensional (1D) disparity value, while circuit 112 may be configured to receive a two-dimensional (2D) optical flow value. In various embodiments, the input data of circuit 112 generally contains horizontal shift information and vertical shift information.

[0039] In an example, each of the neural networks of blocks 102 and 104 may be implemented using a directed acyclic graph (DAG) and corresponding weighted sets 102a and 104a. Figure 3 The unsupervised training process described herein determines the corresponding weighted set. In an example embodiment, block 102 may be implemented as a generative neural network model, and block 104 may be implemented as a discriminative neural network model. In various embodiments, block 102 may be utilized during the training and deployment phases of circuit 100', while block 104 may be utilized only during the training phase. In some embodiments, block 104 may exist, but block 104 is inactive after deployment (e.g., as shown by the dotted line). In some embodiments, block 104 may be omitted from the instance of circuit 100' being deployed. In various embodiments, block 102 may have an input that may receive multiple scaled optical flow maps (e.g., optical flow maps for each scale of an optical flow map pyramid) and an output that may present a fused optical flow map. Block 102 may generate a fused optical flow map based on multiple scaled optical flow maps based on weighted 102a.

[0040] In an example embodiment, the circuit 110 may have a first input that may receive the signal IMAGES and an output that may transmit the calculated optical flow values ​​of the plurality of scaled optical flow maps to the block 102. The output of the block 102 may be coupled to an input of the circuit 112. The block 102 may be configured to transmit the two-dimensional (2D) fused optical flow values ​​to the circuit 112. In an example embodiment, the signal IMAGES may be transmitted to the circuit 110 using one or more memories or buffers (not shown). In another example, the signal IMAGES may also be transmitted to the block 102 to further improve the fused optical flow values ​​generated by the block 102.

[0041] refer to Figure 3 , a block diagram illustrating a training data flow configuration according to an embodiment of the present invention is shown. In various embodiments, two neural networks can be implemented. Neural network 102 can be configured as a generator network, and neural network 104 can be configured as a discriminator network. In various embodiments, the first neural network and the second neural network are typically implemented using similar architectures. In an example, the first neural network and the second neural network can be implemented as a convolutional neural network (CNN) or a deep convolutional neural network (DCNN). However, other network architectures can be implemented to meet the design criteria of a specific application.

[0042] In various embodiments, neural networks 102 and 104 may be connected during a training phase of operation of circuits 100 and 100' to form a generative adversarial network (GAN). Network 102 may receive an image at a first input and receive a plurality of scaled disparity (or optical flow) map values ​​at a second input. Network 102 may be configured to extract features from an image received as an input to network 102 and to generate a fused disparity (or optical flow) map based on the values ​​of the plurality of scaled disparity (or optical flow) maps also received as input. The fused disparity (or optical flow) map generated by network 102 may be used to generate a regenerated image using a warp module 120. In various embodiments, the warp module 120 may generate a regenerated image by performing a warp operation on a first (e.g., left or earlier in time) image based on the fused disparity (or optical flow) map to generate a second (e.g., right or later in time) image. In some embodiments, the warping module 120 may generate a reconstructed image by performing a warping operation on a first (e.g., right or temporally later) image based on a fused disparity (or optical flow) map to generate a second (e.g., left or temporally earlier) image.

[0043] The neural network 104 may have a first input that may receive a regenerated image from the warping module 120, a second input that may receive an original image, a third input that may receive values ​​of a plurality of scaled disparity (or optical flow) maps, and an output that may provide information for modifying the weights 102a utilized in the layers of the neural network 102. During a training phase, the neural network 104 may be configured to indicate the probability that an input image is a regenerated (e.g., right or later in time) image created by the warping module 120 or an originally captured image. During a training phase, the network 104 may take images from a training dataset (e.g., originally captured images) and generated images from the warping module 120. When the network 104 successfully identifies a real image and a generated image, the parameters of the network 104 generally remain unchanged, but the parameters of the network 102 are generally updated to improve the generated image. When the network 102 deceives the network 104, the parameters of the network 102 generally remain unchanged, but the parameters of the network 104 (e.g., weights 104a) are generally updated to improve the classification of the input image. When the predetermined threshold is met, the network 102 generates a copy of the original captured image each time, and the network 104 cannot tell the difference and predicts uncertainty in each case. In an example, the threshold for the original captured image and the generated image can be 50%. However, other thresholds can be implemented to meet the criteria of a specific application. When the predetermined threshold is met, the network 102 can be deployed (e.g., the network 102 is utilized without the network 104 to perform inference operations).

[0044] refer to Figure 4-8 , a block diagram illustrating a data flow for multi-scale disparity fusion according to an example embodiment of the present invention is shown. In various embodiments, the process for implementing multi-scale disparity fusion typically utilizes a one-dimensional (1D) warping engine to generate a reconstructed image.

[0045] refer to Figure 4 , a diagram illustrating a process 200 for generating a multi-scale disparity map is shown. In an example embodiment, a disparity calculator 202 may receive a left image 204 and a right image 206. The disparity calculator 202 may be configured to compare the images to generate a multi-scale disparity map 208. In an example, the disparity calculator 202 may use a pyramid method to calculate disparity values ​​208a-208n at multiple resolutions (or scales). In an example, when the process 200 proceeds to a lower resolution, small changes may be ignored and large changes may be reduced to small changes. The disparity value may be calculated together with the scale.

[0046] refer to Figure 5, a diagram illustrating a process 210 of generating a fused disparity map is shown. In an example embodiment, a disparity fusion generator 212 may be configured to receive a left image 204, a right image 206, and a multi-scale disparity map 208. Based on the left image 204, the right image 206, and the multi-scale disparity map 208, the disparity fusion generator 212 may generate a fused disparity map 214. In an example, the disparity fusion generator 212 may be implemented as an artificial neural network that takes disparity values ​​from each of the multi-scale disparity maps 208 and the left / right images as input and outputs a fused disparity map. In various embodiments, the disparity fusion generator 212 combines the disparity values ​​from each of the multi-scale disparity maps 208 and generates an output that results in a better quality disparity map. The output generally provides a better quality disparity map because (i) computing disparity only in high-resolution images may not generate a dense disparity map due to noise; and (ii) computing disparity only in low-resolution images allows for a denser disparity map, but with lower disparity accuracy. In various embodiments, the disparity fusion generator 212 is implemented using the network 102.

[0047] refer to Figure 6 , a diagram illustrating a process 220 for generating a regenerated image using a fused disparity map 214 is shown. In an example embodiment, a warping operation 222 may be configured to receive a left image 204 and a fused disparity map 214. Based on the fused disparity map 214, the warping operation 222 typically warps the left image 204 to generate a regenerated right image 224. In an example, the warping operation 222 typically implements a 1D warping scheme that takes data from the left image 204 and the fused disparity map 214 as input and generates corresponding right image data, which is presented as an output of the warping operation 222. Typically, the fused disparity map 214 is used to provide pixel shifts in the horizontal direction. In another example, the warping operation 222 may be configured to receive a right image 206 and a fused disparity map 214, and based on the fused disparity map 214, warps the right image 206 to generate a regenerated left image (not shown).

[0048] refer to Figure 7 , a diagram illustrating a classification process 230 according to an example embodiment of the present invention is shown. During a training phase, a discriminator 232 (e.g., network 104) may receive a captured left image 204, a captured right image 206, and a multi-scale disparity map 208. The discriminator 232 may be configured to use features extracted from the input images (e.g., images 204 and 206) and the multi-scale disparity map 208 to determine whether the image is an actual captured image. The discriminator 232 may be trained to validate a TRUE state 234 in response to the input of a captured right image 206.

[0049] refer to Figure 8, a diagram illustrating another example of a classification process 230 according to an example embodiment of the present invention is shown. During the training phase, the discriminator 232 (e.g., the network 104) may also receive the captured left image 204, the regenerated right image 224, and the multi-scale disparity map 208. The discriminator 232 may be configured to use features extracted from the input images (e.g., images 204 and 224) and the multi-scale disparity map 208 to determine whether one of the images is a regenerated image. The discriminator 232 may be trained to assert the FALSE state 236 in response to input of the regenerated right image 224 instead of the captured right image 206.

[0050] refer to Figure 9-13 , shows a block diagram illustrating a data flow for multi-scale optical flow fusion according to an embodiment of the present invention. In various embodiments, multi-scale optical flow fusion utilizes a two-dimensional (2D) warp engine to replace the 1D warp engine used in parallax fusion.

[0051] refer to Fig. 9 , a diagram illustrating a process 300 for generating a multi-scale optical flow map is shown. In an example embodiment, an optical flow calculator 302 may receive a first image frame 304 and a second image frame 306. The first image frame 304 may be from time T, and the second image frame may be from time T+1. The optical flow calculator 302 may be configured to compare images to generate a multi-scale optical flow map 308. In an example, the optical flow calculator 302 may use a pyramid method to calculate optical flow values ​​308a-308n at multiple resolutions (or scales). In an example, when the process 300 proceeds to a lower resolution, small motions may be ignored, and large motions may be reduced to small motions. The process 300 may calculate optical flow as well as scale.

[0052] refer to Fig.10 , a diagram illustrating a process 310 of generating a fused optical flow map is shown. In an example embodiment, an optical flow map fusion generator 312 may be configured to receive a first image frame 304, a second (later) image frame 306, and a multi-scale optical flow map 308. In various embodiments, the optical flow map fusion generator 312 is implemented using the network 102. Based on features extracted from the first image frame 304 and the second (later) image frame 306 and the multi-scale optical flow map 308, the optical flow map fusion generator 312 may generate a fused optical flow map 314.

[0053] In various embodiments, the optical flow map fusion generator 312 is typically implemented as an artificial neural network that takes the optical flow map of each scale and two consecutive images as input and outputs a fused optical flow map 314. Since the optical flow map fusion generator 312 combines the optical flow maps of each scale together, the fused optical flow map 314 generated by the optical flow map fusion generator 312 is typically of better quality than an optical flow map generated using conventional techniques.

[0054] refer to Fig.11 , a diagram illustrating a process 320 of generating a regenerated image using a fused optical flow map 314 is shown. In an example embodiment, a 2D warping operation 322 may be configured to receive a first image frame 304 and a fused optical flow map 314. Based on the fused optical flow map 314, the 2D warping operation 322 typically warps the first (earlier) image frame 304 to generate a regenerated later image frame 324 (e.g., frame T+1). In various embodiments, the 2D warping operation 322 takes a previous image and a fused optical flow map 314 as input, and outputs (generates) a corresponding next frame of image. In an example, the fused optical flow map 314 may be used to specify pixel shifts in the horizontal and vertical directions.

[0055] refer to Fig.12 , a diagram illustrating a classification process 330 according to an example embodiment of the present invention is shown. During a training phase, a discriminator 332 (e.g., network 104) may receive a temporally earlier image frame 304, a temporally later image frame 306, and a multi-scale optical flow map 308. The discriminator 332 may be configured to use features extracted from input images (e.g., images 304 and 306) and the multi-scale disparity map 308 to determine whether one of the images is a temporally later image frame. The discriminator 332 may be trained to validate a TRUE state 334 in response to an input of a temporally later image frame 306.

[0056] refer to Fig.13 , a diagram illustrating another example of a classification process 330 according to an example embodiment of the present invention is shown. During a training phase, a discriminator 332 (e.g., network 104) may receive a temporally earlier image frame 304, a regenerated temporally later image frame 324, and a multi-scale optical flow map 308. The discriminator 332 may be trained to assert a FALSE state 336 in response to an input image being a regenerated temporally later image frame 324 rather than a captured temporally later image frame 306.

[0057] refer to Fig.14 , a diagram illustrating a context in which an unsupervised training process according to an example embodiment of the present invention may be implemented. In the example, according to the above combined Figure 3In the example embodiment of the invention described, the system 600 may be configured (e.g., by software) to perform an unsupervised training process. In the example, the system 600 may include a computer 602 and a computer-readable storage medium 604. In the example, the computer 602 may be implemented as a general-purpose computer system. In the example, the computer-readable storage medium 604 may include non-volatile media, including but not limited to magnetic storage media (e.g., a hard disk drive or HDD), optical storage media (e.g., a compact disk (CD), a digital video disk (DVD), a Blu-ray disk (BD)), a solid-state drive (SSD), flash memory, and / or a network-associated storage system (NAS). In the example, the computer 602 and the computer-readable storage medium 604 may be coupled together to exchange programs and data. In the example, the implementation Figure 3 The program (or programs) 606 of the unsupervised training process shown in the figure can be stored on the computer-readable storage medium 604 or in the cloud-based resources 608. In the example, the computer 602 can also be configured to perform the unsupervised training process according to the example embodiment of the present invention using the cloud resources 608. In the example, the computer 602 can be configured to execute the program (or programs) 606 of the unsupervised training process according to the example embodiment of the present invention via one or more application program interfaces (APIs).

[0058] In the example, the computer 602 may include, but is not limited to, a processor 610, a memory 612, a display 614, and a user interface 616. In various embodiments, the processor 610 may include, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), and a video processing unit (VPU). In various embodiments, the memory 612 may include, but is not limited to, a random access memory (e.g., SRAM, DRAM, FLASH, etc.), a read-only memory (ROM), and a cache memory. The display 614 and the user interface 616 generally allow the user to start and monitor the computer 602 during the training and / or quantization phase to execute a program (or multiple programs) 606 that implements an unsupervised training process according to an example embodiment of the present invention.

[0059] refer to Fig.15, a diagram of a camera system 900 is shown, which illustrates an example implementation of a computer vision system in which a multi-scale disparity / optical flow fusion algorithm according to an example embodiment of the present invention can be implemented. In one example, the electronic devices of the camera system 900 can be implemented as one or more integrated circuits. In an example, the camera system 900 can be built around a processor / camera chip (or circuit) 902. In an example, the processor / camera chip 902 can be implemented as an application-specific integrated circuit (ASIC) or a system on a chip (SOC). The processor / camera circuit 902 is typically combined with hardware and / or software / firmware that can be configured to implement the above combination Figures 1 to 14 Describe the circuit and process.

[0060] In an example, the processor / camera circuit 902 may be connected to a lens and sensor assembly 904. In some embodiments, the lens and sensor assembly 904 may be a component of the processor / camera circuit 902 (e.g., a SoC component). In some embodiments, the lens and sensor assembly 904 may be a separate component from the processor / camera circuit 902 (e.g., the lens and sensor assembly may be an interchangeable component compatible with the processor / camera circuit 902). In some embodiments, the lens and sensor assembly 904 may be part of a separate camera connected to the processor / camera circuit 902 (e.g., via a video cable, a high-definition media interface (HDMI) cable, a universal serial bus (USB) cable, an Ethernet cable, or a wireless link).

[0061] The lens and sensor assembly 904 may include a block (or circuit) 906 and / or a block (or circuit) 908. Circuit 906 may be associated with the lens assembly. Circuit 908 may be implemented as one or more image sensors. In one example, circuit 908 may be implemented as a single sensor. In another example, circuit 908 may be implemented as a stereo pair of sensors. The lens and sensor assembly 904 may include other components (not shown). The number, type, and / or function of the components of the lens and sensor assembly 904 may vary according to the design criteria of a particular implementation.

[0062] The lens assembly 906 can capture and / or focus light input received from the environment surrounding the camera system 900. The lens assembly 906 can capture and / or focus light for (multiple) image sensors 908. The lens assembly 906 can implement one or more optical lenses. The lens assembly 906 can provide a zoom feature and / or a focusing feature. The lens assembly 906 can be implemented with additional circuitry (e.g., a motor) to adjust the direction, zoom, and / or aperture of the lens assembly 906. The lens assembly 906 can be oriented, tilted, translated, zoomed, and / or rotated to provide a target view of the environment surrounding the camera system 900.

[0063] Image sensor(s) 908 may receive light from lens assembly 906. Image sensor(s) 908 may be configured to convert the received focused light into digital data (e.g., a bit stream). In some embodiments, image sensor(s) 908 may perform analog-to-digital conversion. For example, image sensor(s) 908 may perform photoelectric conversion on the focused light received from lens assembly 906. Image sensor(s) 908 may present the converted image data as a bit stream in a color filter array (CFA) format. Processor / camera circuitry 902 may convert the bit stream into video data, a video file, and / or a video frame (e.g., human-readable content).

[0064] The processor / camera circuit 902 may also be connected to: (i) optional audio input / output circuitry including an audio codec 910, a microphone 912, and a speaker 914; (ii) a memory 916, which may include dynamic random access memory (DRAM); (iii) non-volatile memory (e.g., NAND flash memory) 918, removable media (e.g., SD, SDXC, etc.) 920, one or more serial (e.g., RS-485, RS-232, etc.) devices 922, one or more universal serial bus (USB) devices (e.g., a USB host) 924, and a wireless communication device 926.

[0065] In various embodiments, the processor / camera circuit 902 may include a plurality of blocks (or circuits) 930a-930n, a plurality of blocks (or circuits) 932a-932n, a block (or circuit) 934, a block (or circuit) 936, a block (or circuit) 938, a block (or circuit) 940, a block (or circuit) 942, a block (or circuit) 944, a block (or circuit) 946, a block (or circuit) 948, a block (or circuit) 950, a block (or circuit) 952, and / or a block (or circuit) 954. The plurality of circuits 930a-930n may be processor circuits. In various embodiments, the circuits 930a-930n may include one or more embedded processors (e.g., ARM, etc.). The circuits 932a-932n may implement a plurality of computer vision-related processor circuits. In an example, one or more of the circuits 932a-932n may implement various computer vision-related applications. The circuit 934 may be a digital signal processing (DSP) module. In some embodiments, circuit 934 may implement separate image DSP modules and video DSP modules.

[0066] Circuit 936 may be a storage interface. Circuit 936 may interface processor / camera circuit 902 with DRAM 916, non-volatile memory 918, and removable media 920. One or more of DRAM 916, non-volatile memory 918, and / or removable media 920 may store computer-readable instructions. The computer-readable instructions may be read and executed by processors 930a-930n. In response to the computer-readable instructions, processors 930a-930n may be operable to act as controllers for processors 932a-932n. For example, the resources of processors 932a-932n may be configured to efficiently perform various specific operations in hardware, and processors 930a-930n may be configured to make decisions about how to process inputs / outputs to / from various resources of processor 932.

[0067] Circuit 938 may implement a local memory system. In some embodiments, local memory system 938 may include, but is not limited to, a cache memory (e.g., L2CACHE), a direct memory access (DMA) engine, a graphics direct memory access (GDMA) engine, and a fast random access memory. In an example, DAG memory 968 may be implemented in local memory system 938. Circuit 940 may implement a sensor input (or interface). Circuit 942 may implement one or more control interfaces, including but not limited to an inter-device communication (IDC) interface, an inter-integrated circuit (I2C) interface, a serial peripheral interface (SPI), and a pulse width modulation (PWM) interface. Circuit 944 may implement an audio interface (e.g., an I2S interface, etc.). Circuit 946 may implement a clock circuit, including but not limited to a real-time clock (RTC), a watchdog timer (WDT), and / or one or more programmable timers. Circuit 948 may implement an input / output (I / O) interface. Circuit 950 may be a video output module. Circuit 952 may be a communication module. Circuit 954 may be a security module. Circuits 930 - 954 may be connected to each other using one or more buses, interfaces, traces, protocols, or the like.

[0068] Circuit 918 may be implemented as a non-volatile memory (e.g., NAND flash memory, NOR flash memory, etc.). Circuit 920 may include one or more removable media cards (e.g., secure digital media (SD), secure digital extended capacity media (SDXC), etc.). Circuit 922 may include one or more serial interfaces (e.g., RS-485, RS-232, etc.). Circuit 924 may be an interface for connecting to a universal serial bus (USB) host or serving as a USB host. Circuit 926 may be a wireless interface for communicating with a user device (e.g., a smart phone, a computer, a tablet computing device, a cloud resource, etc.). In various embodiments, circuits 904-926 may be implemented as components external to the processor / camera circuit 902. In some embodiments, circuits 904-926 may be on-board components of the processor / camera circuit 902.

[0069] The control interface 942 may be configured to generate signals (e.g., IDC / I2C, STEPPER, IRIS, AF / ZOOM / TILT / PAN, etc.) for controlling the lens and sensor assembly 904. The signal IRIS may be configured to adjust the aperture of the lens assembly 906. The interface 942 may enable the processor / camera circuit 902 to control the lens and sensor assembly 904.

[0070] The storage interface 936 may be configured to manage one or more types of storage and / or data access. In one example, the storage interface 936 may implement a direct memory access (DMA) engine and / or a graphics direct memory access (GDMA). In another example, the storage interface 936 may implement a secure digital (SD) card interface (e.g., to connect to the removable media 920). In various embodiments, programming code (e.g., executable instructions for controlling various processors and encoders of the processor / camera circuit 902) may be stored in one or more of the memories (e.g., DRAM 916, NAND 918, etc.). The programming code, when executed by one or more of the processors 930, typically causes one or more components in the processor / camera circuit 902 to configure video synchronization operations and begin video frame processing operations. The resulting compressed video signal may be presented to the storage interface 936, the video output 950, and / or the communication interface 952. The storage interface 936 can transfer program codes and / or data between external media (eg, DRAM 916 , NAND 918 , removable media 920 , etc.) and a local (internal) memory system 938 .

[0071] The sensor input 940 may be configured to send data to / receive data from the image sensor 908. In one example, the sensor input 940 may include an image sensor input interface. The sensor input 940 may be configured to send a captured image (e.g., picture elements, pixels, data) from the image sensor 908 to the DSP module 934, one or more of the processors 930, and / or one or more of the processors 932. The data received by the sensor input 940 may be used by the DSP 934 to determine the brightness (Y) and chrominance (U and V) values ​​from the image sensor 908. The sensor input 940 may provide an interface to the lens and sensor assembly 904. The sensor input interface 940 may enable the processor / camera circuit 902 to capture image data from the lens and sensor assembly 904.

[0072] The audio interface 944 may be configured to send / receive audio data. In one example, the audio interface 944 may implement an audio inter-IC sound (I2S) interface. The audio interface 944 may be configured to send / receive data in a format implemented by the audio codec 910.

[0073] The DSP module 934 may be configured to process digital signals. The DSP module 934 may include an image digital signal processor (IDSP), a video digital signal processor DSP (VDSP), and / or an audio digital signal processor (ADSP). The DSP module 934 may be configured to receive information from the sensor input 940 (e.g., pixel data values ​​captured by the image sensor 908). The DSP module 934 may be configured to determine pixel values ​​(e.g., RGB, YUV, brightness, chrominance, etc.) based on the information received from the sensor input 940. The DSP module 934 may be further configured to support or provide a sensor RGB to YUV raw image pipeline to improve image quality, perform bad pixel detection and correction, demosaicing, white balance, color and tone correction, gamma correction, hue adjustment, saturation, brightness and contrast adjustment, chroma and brightness noise filtering.

[0074] The I / O interface 948 may be configured to send / receive data. The data sent / received by the I / O interface 948 may be miscellaneous information and / or control data. In one example, the I / O interface 948 may implement one or more of a general purpose input / output (GPIO) interface, an analog-to-digital converter (ADC) module, a digital-to-analog converter (DAC) module, an infrared (IR) remote interface, a pulse width modulation (PWM) module, a universal asynchronous receiver transmitter (UART), an infrared (IR) remote interface, and / or one or more synchronous data communication interfaces (IDC SPI / SSI).

[0075] The video output module 950 can be configured to send video data. For example, the processor / camera circuit 902 can be connected to an external device (e.g., a television, a monitor, a laptop computer, a tablet computing device, etc.). The video output module 950 can implement a high-definition multimedia interface (HDMI), a PAL / NTSC interface, an LCD / TV / parallel interface, and / or a DisplayPort interface.

[0076] The communication module 952 may be configured to send / receive data. The data sent / received by the communication module 952 may be transmitted / received according to a specific protocol (e.g., USB, Wi-Fi, UART, etc.). In one example, the communication module 952 can implement a secure digital input output (SDIO) interface. The communication module 952 can include a communication interface that receives and sends data over one or more wireless protocols (e.g., The communication module 952 may also include support for wireless communications using one or more universal serial bus protocols (e.g., USB 1.0, 2.0, 3.0, etc.). The processor / camera circuit 902 may also be configured to be powered via a USB connection. However, other communication and / or power interfaces may be implemented accordingly to meet the design criteria of a particular application.

[0077] The security module 954 may include a set of advanced security features to achieve advanced on-device physical security, including OTP, secure boot, As well as I / O visualization and DRAM scrambling. In an example, the security module 958 can include a true random number generator. In an example, the security module 954 can be used for DRAM communication encryption on the processor / camera circuit 902.

[0078] The processor / camera circuit 902 can be configured (e.g., programmed) to control one or more lens assemblies 906 and one or more image sensors 908. The processor / camera circuit 902 can receive raw image data from (multiple) image sensors 908. The processor / camera circuit 902 can simultaneously (in parallel) encode the raw image data into multiple encoded video streams. The multiple video streams can have various resolutions (e.g., VGA, WVGA, QVGA, SD, HD, Ultra HD, 4K, etc.). The processor / camera circuit 902 can receive encoded and / or unencoded (e.g., raw) audio data at the audio interface 944. The processor / camera circuit 902 can also receive encoded audio data from the communication interface 952 (e.g., USB and / or SDIO). The processor / camera circuit 902 can provide the encoded video data to the wireless interface 926 (e.g., using a USB host interface). The wireless interface 926 can include support for one or more wireless and / or cellular protocols (e.g., The processor / camera circuit 902 may also include support for wireless communications using one or more universal serial bus protocols (e.g., USB 1.0, 2.0, 3.0, etc.).

[0079] refer to Fig.16 , a diagram illustrating a processing circuit 902 in which a context of a disparity / optical flow map fusion network according to an example embodiment of the present invention can be implemented is shown. In various embodiments, the processing circuit 902 can be implemented as part of a computer vision system. In various embodiments, the processing circuit 902 can be implemented as part of a camera, a computer, a server (e.g., a cloud server), a smart phone (e.g., a cellular phone), a personal digital assistant, etc. In an example, the processing circuit 902 can be configured for applications including, but not limited to, autonomous and semi-autonomous vehicles (e.g., cars, trucks, motorcycles, agricultural machinery, drones, airplanes, etc.), manufacturing and / or security and surveillance systems. In contrast to a general-purpose computer, the processing circuit 902 typically includes hardware circuits that are optimized to provide high-performance image processing and computer vision pipelines in a minimal area and at minimal power consumption. In an example, hardware modules designed to reduce computational complexity and use resources efficiently can be used to implement various operations for performing image processing, feature detection / extraction, and / or object detection / classification for computer (or machine) vision.

[0080] In an example embodiment, the processing circuit 902 may include a block (or circuit) 930i, a block (or circuit) 932i, a block (or circuit) 916, and / or a memory bus 917. The circuit 930i may implement a first processor. The circuit 932i may implement a second processor. In an example, the circuit 932i may implement a computer vision processor. In an example, the processor 932i may be an intelligent vision processor. The circuit 916 may implement an external memory (e.g., a memory external to the circuits 930i and 932i). In an example, the circuit 916 may be implemented as a dynamic random access memory (DRAM) circuit. The processing circuit 902 may include other components (not shown). The number, type, and / or arrangement of the components of the processing circuit 902 may vary according to the design criteria of a particular implementation.

[0081] Circuit 930i can implement a processor circuit. In some embodiments, a general-purpose processor circuit can be used to implement processor circuit 930i. Processor 930i can be operable to interact with circuit 932i and circuit 916 to perform various processing tasks. In an example, processor 930i can be configured as a controller for circuit 932i. Processor 930i can be configured to execute computer-readable instructions. In one example, computer-readable instructions can be stored by circuit 916. In some embodiments, computer-readable instructions can include controller operations. Processor 930i can be configured to communicate with circuit 932i and / or access results generated by components of circuit 932i. In an example, processor 930i can be configured to utilize circuit 932i to perform operations associated with one or more neural network models.

[0082] In an example, the processor 930i may be configured to program the circuit 932i with one or more pre-trained artificial neural network models (ANNs), the ANNs including a region proposal network (RPN) 100, a region-based convolutional neural network (RCNN) 102, and corresponding weights / kernels (WGTS) 104 associated with the RPN 100 and the RCNN 102. In various embodiments, the RPN 100 and the RCNN 102 may be configured (trained and / or quantized) for operation in an edge device. In an example, the processing circuit 902 may be coupled to a sensor (e.g., a video camera, etc.) configured to generate data inputs. The processing circuit 902 may be configured to generate one or more outputs in response to data inputs from the sensor based on one or more inferences made by executing the RPN 100 and the RCNN 102 using the corresponding weights / kernels (WGTS) 104. The operations performed by the processor 930i may vary according to the design criteria of a particular implementation.

[0083] In various embodiments, circuit 916 may implement dynamic random access memory (DRAM) circuits. Circuit 916 is generally operable to store multi-dimensional arrays of input data elements and various forms of output data elements. Circuit 916 may exchange input data elements and output data elements with processor 930i and processor 932i.

[0084] Processor 932i can implement computer vision processor circuit. In an example, processor 932i can be configured to implement various functions for computer vision. Processor 932i is generally operable to perform specific processing tasks arranged by processor 930i. In various embodiments, all or part of processor 932i can be implemented only in hardware. Processor 932i can directly execute data streams generated by software (e.g., directed acyclic graph, etc.) for multi-scale disparity / optical flow fusion and specified processing (e.g., computer vision) tasks. In some embodiments, processor 932i can be a representative example of several computer vision processors implemented by processing circuit 902 and configured to operate together.

[0085] In an example embodiment, processor 932i generally includes block (or circuit) 960, one or more blocks (or circuits) 962a-962n, block (or circuit) 960, path 966 and block (or circuit) 968. Block 960 can implement a scheduler circuit. Blocks 962a-962n can implement hardware resources (or engines). Block 964 can implement a shared memory circuit. Block 968 can implement a directed acyclic graph (DAG) memory. In an example embodiment, one or more of circuits 962a-962n can include blocks (or circuits) 970a-970n. In the example shown, circuits 970a and 970b are implemented.

[0086] In an example embodiment, circuit 970a may implement a convolution operation. In another example, circuit 970b may be configured to provide a pooling operation. Circuits 970a and 970b may be utilized to provide multi-scale disparity / optical flow fusion according to an example embodiment of the present invention. Convolution operations and pooling operations may be used to perform computer (or machine) vision tasks (e.g., as part of an object detection process, etc.). In yet another example, one or more of circuits 962c-962n may include blocks (or circuits) 970c-970n (not shown) to provide multi-dimensional convolution calculations.

[0087] In an example, the circuit 932i may be configured to receive a directed acyclic graph (DAG) from the processor 930i. The DAG received from the processor 930i may be stored in a DAG memory 968. The circuit 932i may be configured to execute the DAG for the generator neural network 102 and / or the discriminator neural network 104 using circuits 960, 962a-962n, and 964.

[0088] Multiple signals (e.g., OP_A to OP_N) may be exchanged between circuit 960 and corresponding circuits 962a-962n. Each signal OP_A to OP_N may convey information about performing an operation and / or generating an operation. Multiple signals (e.g., MEM_A to MEM_N) may be exchanged between corresponding circuits 962a-962n and circuit 964. Signals MEM_A to MEM_N may carry data. Signals (e.g., DRAM) may be exchanged between circuit 916 and circuit 964. Signal DRAM may transmit data between circuits 916 and 960 (e.g., on memory bus 966).

[0089] Circuit 960 may implement a scheduler circuit. Scheduler circuit 960 is generally operable to schedule tasks between circuits 962a-962n to perform various computer vision-related tasks defined by processor 930i. Scheduler circuit 960 may assign individual tasks to circuits 962a-962n. Scheduler circuit 960 may assign individual tasks in response to parsing a directed acyclic graph (DAG) provided by processor 930i. Scheduler circuit 960 may time-multiplex tasks to circuits 962a-962n based on the availability of circuits 962a-962n to perform work.

[0090] Each circuit 962a-962n can implement a processing resource (or hardware engine). The hardware engine 962a-962n is generally operable to perform a specific processing task. The hardware engine 962a-962n can be implemented to include a dedicated hardware circuit that is optimized to have high performance and low power consumption when performing a specific processing task. In some configurations, the hardware engines 962a-962n can operate in parallel and independently of each other. In other configurations, the hardware engines 962a-962n can operate together with each other to perform assigned tasks.

[0091] The hardware engines 962a-962n may be homogeneous processing resources (e.g., all circuits 962a-962n may have the same capabilities) or heterogeneous processing resources (e.g., two or more circuits 962a-962n may have different capabilities). The hardware engines 962a-962n are generally configured to execute operators, which may include, but are not limited to, resampling operators, warp operators, component operators that manipulate lists of components (e.g., components may be regions of vectors that share common properties and may be grouped together with bounding boxes), matrix inverse operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, min-max-reduce operators, pooling operators, non-min-non-max suppression operators, aggregation operators, scattering operators, statistical operators, classifier operators, integral image operators, upsampling operators, and downsampling operators of powers of two, among others.

[0092] In various embodiments, hardware engines 962a-962n may be implemented as hardware circuits only. In some embodiments, hardware engines 962a-962n may be implemented as general engines that may be configured to operate as a dedicated machine (or engine) by circuit customization and / or software / firmware. In some embodiments, hardware engines 962a-962n may be implemented alternatively as one or more instances or threads of program code executed on processor 930i and / or one or more processors 932i (including but not limited to vector processors, central processing units (CPUs), digital signal processors (DSPs), or graphics processing units (GPUs)). In some embodiments, scheduler 960 may select one or more of hardware engines 962a-962n for a particular process and / or thread. Scheduler 960 may be configured to assign hardware engines 962a-962n to a particular task in response to parsing a directed acyclic graph stored in DAG memory 968.

[0093] Circuit 964 can implement shared memory circuit. Shared memory 964 can be configured to store data in response to input requests and / or present data in response to output requests (e.g., requests from processor 930i, DRAM 916, scheduler circuit 960, and / or hardware engines 962a-962n). In an example, shared memory circuit 964 can implement on-chip memory for computer vision processor 932i. Shared memory 964 is generally operable to store all or part of a multidimensional array (or vector) of input data elements and output data elements generated and / or utilized by hardware engines 962a-962n. Input data elements can be transferred from DRAM circuit 916 to shared memory 964 via memory bus 917. Output data elements can be sent from shared memory 964 to DRAM circuit 916 via memory bus 917.

[0094] Path 966 may implement a transfer path internal to processor 932 i. Transfer path 966 is generally operable to move data from scheduler circuit 960 to shared memory 964. Transfer path 966 may also be operable to move data from shared memory 964 to scheduler circuit 960.

[0095] Processor 930i is shown as communicating with computer vision processor 932i. Processor 930i can be configured as a controller for computer vision processor 932i. In some embodiments, processor 930i can be configured to transmit instructions to scheduler 960. For example, processor 930i can provide one or more directed acyclic graphs to scheduler 960 via DAG memory 968. Scheduler 960 can initialize and / or configure hardware engines 962a-962n in response to parsing directed acyclic graphs. In some embodiments, processor 930i can receive state information from scheduler 960. For example, scheduler 960 can provide state information and / or readiness of output from hardware engines 962a-962n to processor 930i so that processor 930i can determine one or more next instructions to be executed and / or decisions to be made. In some embodiments, processor 930i can be configured to communicate with shared memory 964 (e.g., directly or through scheduler 960, which receives data from shared memory 964 via path 966). Processor 930i can be configured to retrieve information from shared memory 964 to make decisions. The instructions executed by processor 930i in response to information from computer vision processor 932i can vary according to the design criteria of a particular implementation.

[0096] Circuit 970a can implement a convolution circuit. Convolution circuit 970a can communicate with memory 964 to receive input data and present output data. Convolution circuit 970a is generally operable to grab multiple data vectors from shared memory circuit 964. Each data vector can include multiple data values. Convolution circuit 970a can also be operable to grab a kernel from shared memory 964. The kernel typically includes multiple kernel values. Convolution circuit 970a can also be operable to grab a block from shared memory 964 to an internal (or local) buffer. The block typically includes multiple input tiles. Each input tile can include multiple input values ​​in multiple dimensions. Convolution circuit 970a can also be operable to calculate multiple intermediate values ​​in parallel by multiplying each input tile in the internal buffer with a corresponding one of the kernel values, and calculating an output tile including multiple output values ​​based on the intermediate value. In various embodiments, convolution circuit 970a can be implemented only in hardware. An example of a convolutional computation scheme that can be used to implement circuit 970a can be found in U.S. Pat. No. 10,210,768, which is incorporated herein by reference in its entirety. Circuit 970b can implement a pooling process. In various embodiments, a multi-scale disparity / optical flow fusion scheme according to an embodiment of the present invention can be performed according to the implementation description provided herein.

[0097] refer to Fig.17 , shows the description Fig.16 FIG. 9 is a diagram of an example implementation of a general hardware engine 962x of the present invention. Hardware engine 962x may represent hardware engines 962a-962n. Hardware engine 962x generally includes block (or circuit) 980, block (or circuit) 982, block (or circuit) 984, and multiple blocks (or circuits) 986a-986n. Circuit 980 may be implemented as a memory (or buffer) pair 980a and 980b. Circuit 982 may implement a controller circuit. In an example, circuit 982 may include one or more finite state machines (FSMs) configured to control various operators implemented by hardware engine 962x. Circuit 984 may implement a processing pipeline of hardware engine 962x. Circuits 986a-986n may implement a first-in, first-out (FIFO) memory. Circuits 986a-986n may be configured as input buffers for processing pipeline 984. Shared memory 964 may be configured (eg, by signals from circuitry 982 ) as a plurality of shared input buffers 988a - 988n and one or more output buffers 990 .

[0098] Signals (e.g., ADDR / CONFIG) may be generated by scheduler circuit 960 and received by hardware engine 962x. Signal ADDR / CONFIG may carry address information and configuration data. Signals (e.g., BUSY_LEVEL) may be generated by circuit 982 and transmitted to scheduler circuit 960. Signal BUSY_LEVEL may convey the busy level of hardware engine 962x. Signals (e.g., STATUS / TARGETS) may be generated by circuit 982 and transmitted to scheduler circuit 960. Signal STATUS / TARGETS may provide status information about hardware engine 962x and target information for operands.

[0099] In an example embodiment, buffers 980a and 980b may be configured as dual-bank configuration buffers. The dual-bank buffers may be operable to store configuration information for a currently running operation in one buffer (e.g., buffer 980b) and move configuration information for a next operation to another buffer (e.g., buffer 980a). Scheduler 960 typically loads operator configuration information (including status words in the case where an operator has been partially processed in a previous operator block) into the dual-bank buffers. Once circuit 982 has completed configuration information for a running operation and has received configuration information for a next operation, buffers 980a and 980b may be swapped.

[0100] Circuit 982 typically implements control circuitry for hardware engine 962x. Circuit 982 determines when to switch from a currently running operator to a new operator. Controller 982 is typically operable to control the movement of information into and out of hardware engine 982x and within hardware engine 982x. Typically, the operation of hardware engine 962x is pipelined. During an operator switch, the front end of pipeline 984 may already be processing data for the new operator while the tail end of pipeline 984 is still completing processing associated with the old operator.

[0101] The circuit 984 may implement pipeline circuitry. The pipeline circuit 984 is generally operable to process operands received from the shared memory 964 using functions designed into the hardware engine 962x. The circuit 984 may transfer data generated by the executed functions to one or more shared buffers 990.

[0102] Buffers 986a-986n may implement FIFO buffers. FIFO buffers 986a-986n may be operable to store operands received from shared buffers 988a-988n for processing in pipeline 984. In general, the number of FIFO buffers and the number of shared buffers implemented may be varied to meet the design criteria of a particular application.

[0103] As will be appreciated by those skilled in the relevant art, one or more of conventional general purpose processors, digital computers, microprocessors, microcontrollers, RISC (Reduced Instruction Set Computer) processors, CISC (Complex Instruction Set Computer) processors, SIMD (Single Instruction Multiple Data) processors, signal processors, central processing units (CPUs), arithmetic logic units (ALUs), video digital signal processors (VDSPs), distributed computer resources, and / or similar computing machines programmed according to the teachings of this specification may be used to design, model, simulate, and / or emulate the system. Figure 1-17 The functions performed by the diagrams and the structures shown. As will be apparent to those skilled in the relevant art, a skilled programmer can readily prepare appropriate software, firmware, coding, routines, instructions, opcodes, microcodes and / or program modules based on the teachings of the present disclosure. The software is typically embodied in one or more media (e.g., non-transitory storage media) and can be executed sequentially or in parallel by one or more of the processors.

[0104] Embodiments of the present invention may also be implemented in one or more of an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), a PLD (programmable logic device), a CPLD (complex programmable logic device), a sea of ​​gates, an ASSP (application specific standard product), and an integrated circuit. The circuit may be implemented based on one or more hardware description languages. Embodiments of the present invention may be utilized in conjunction with flash memory, non-volatile memory, random access memory, read-only memory, magnetic disk, floppy disk, optical disk such as DVD and DVD RAM, magneto-optical disk, and / or distributed storage systems.

[0105] When used in conjunction with "is" and a verb herein, the terms "may" and "generally" are intended to convey the intention that the description is exemplary and is considered broad enough to cover both the specific examples presented in the present disclosure and alternative examples that can be derived based on the present disclosure. The terms "may" and "generally" as used herein should not be interpreted as necessarily implying the desirability or possibility of omitting the corresponding element.

[0106] While the invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the invention.

Claims

1. A device for parallax / optical flow fusion, comprising: an interface configured to receive an image from a capture device; as well as A processor configured to: (i) extract features from the image; (ii) generating a fused disparity map in response to the extracted features; (iii) generating a reconstructed image by performing a warp on a first subset of the images based on (a) the fused disparity map and (b) a first parameter; (iv) performing a comparison between a sample image and an image in the first subset of the images; (v) performing classification of the sample image based on a second parameter; and (vi) updating the first parameter and the second parameter in response to whether the classification is correct, wherein (a) the classification includes indicating whether the sample image is one of the images in the second subset of images or one of the regenerated images; and (b) the regenerated images are used as a training data set.

2. The device according to claim 1, wherein: (i) the capture device comprises a first camera and a second camera configured as a stereo camera; (ii) the first subset of images is captured by the first camera; And (iii) the second subset of the images is captured by the second camera.

3. The device according to claim 2, wherein: (i) the first subset of images includes left images; (ii) the second subset of images includes right images; and (iii) the training data set includes data for disparity calculation.

4. The device according to claim 1, wherein: (i) the first subset of images includes the images captured at a first time in the sequence of images; (ii) the second subset of images includes the images captured at a second time in the sequence of images; and (iii) the training dataset includes optical flow information, wherein the second time of the second subset of images is later than the first time compared to the first subset of images.

5. The device according to claim 1, wherein: The sample image is randomly selected from the second subset of images or the reconstructed image.

6. The device according to claim 5, wherein: (a) updating the first parameter to adjust the reproduced image so as to reduce the probability that the classification of the sample image is correct; and (b) updating the second parameter to increase the probability that the classification of the sample image is correct.

7. The device according to claim 6, wherein: When the probability is greater than a threshold amount, the reconstructed image is used as the training data set.

8. The device according to claim 7, wherein: The threshold amount is 50%.

9. The device according to claim 1, wherein: The processor is configured to implement: (i) a first neural network to implement a first model based on the first parameters; and (ii) a second neural network to implement a second model based on the second parameters.

10. The device according to claim 9, wherein: The first neural network is configured to: (a) extract the features from the first subset of the images and the second subset of the images; (b) generating the fused disparity map in response to the extracted features; and (c) generating the reconstructed image; and The second neural network is configured to perform: (a) said comparison of said sample image with an image in said first subset of said images; and (b) said classification of said sample image based on said second parameter.

11. The device according to claim 9, wherein: (i) the first neural network implements a generative neural network model; and (ii) the second neural network implements a discriminative neural network model.

12. The device according to claim 11, wherein: The generative neural network model and the discriminative neural network model are configured as an unsupervised generative adversarial network.

13. The device according to claim 11, wherein: The processor is configured to operate in a training mode of operation using the generative neural network model and the discriminative neural network model to generate the training data set.

14. The device according to claim 13, wherein: The processor is also configured to operate in a data generation mode of operation using the generative neural network model and to disable the discriminative neural network model.

15. The device according to claim 14, wherein: After the generative neural network model has been sufficiently trained during the training mode of operation, the processor operates in the data generation mode of operation.

Images