Method of analyzing an object, image processing system and computer readable storage medium

Abstract

The invention relates to a method of analyzing objects, an image processing system and a computer readable storage medium. The method comprises segmenting frames and matching at least one object in a first frame with a corresponding object in a second frame. The method optionally comprises estimating a motion of the at least one matched object between the frames. Also disclosed is a method of generating a training dataset suitable for training a machine learning algorithm to estimate a motion of an object. Processing systems configured to perform these methods are also provided.

Description

Technical Field

[0001] This invention relates to methods for analyzing objects, image processing systems, and computer-readable storage media. Background Technology

[0002] Analyzing the behavior of objects between frames is a recurring task in vision applications. In particular, it is often desirable to estimate the motion of one or more objects in a scene between the first and second frames.

[0003] The goal of motion estimation is to determine how pixels move from a reference frame to a target frame. Several methods have been proposed to address this task, some of which utilize deep neural networks (DNNs). Early DNN-based methods for motion estimation, such as FlowNet, are examples.

[0004] While struggling to compete with established classical methods, DNN-based approaches have recently seen significant performance improvements. State-of-the-art DNN-based methods, including PwC-Net (“PwC-net: CNNs for optical flow using pyramid, warping, and cost volume” by D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, Proceedings of the 2018 IEEE International Conference on Pattern Recognition in Computer Vision, pp. 8934–8943), have begun to outperform classical methods in both computational efficiency and accuracy. DNN-based methods represent a new way of thinking about motion estimation problems and have been gradually improving over time. Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, a series of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] A method for analyzing objects in a first frame and a second frame is disclosed. The method includes: segmenting the frame; and matching at least one object in the first frame with a corresponding object in the second frame. Optionally, the method includes estimating the motion of at least one matched object between frames. A method for generating a training dataset suitable for training a machine learning algorithm to estimate the motion of objects is also disclosed.

[0007] It also provides a processing system that is configured to execute these methods.

[0008] According to a first aspect, a method is provided for analyzing one or more objects in a set of frames including at least a first frame and a second frame, the method comprising: The first frame is segmented to generate multiple first masks, each first mask identifying a pixel belonging to a potential object instance detected in the first frame; For each potential object instance detected in the first frame, extract the first feature vector representing the potential object instance from the first frame; The second frame is segmented to generate multiple second masks, each second mask identifying a pixel belonging to a potential object instance detected in the second frame; For each potential object instance detected in the second frame, extract a second feature vector representing the potential object instance from the second frame; and At least in part based on a first feature vector, a first mask, a second feature vector, and a second mask, at least one of the potential object instances in the first frame is matched with one of the potential object instances in the second frame.

[0009] Each potential object instance can exist in a corresponding region of interest within a frame, which is defined by a bounded box.

[0010] Segmenting the first frame to generate the first mask may include: identifying regions of interest in the first frame and, for each region of interest, segmenting pixels within a relevant bounded box to generate the corresponding first mask; and segmenting the second frame to generate the second mask may include: identifying regions of interest in the second frame and, for each region of interest, segmenting pixels within a relevant bounded box to generate the corresponding second mask.

[0011] Segmentation for each potential object instance can be based solely on those pixels within the bounded box. Segmentation can also include refining the bounded box. Machine learning algorithms can be used to generate and later refine the bounded boxes.

[0012] Matching may include clustering potential object instances detected in the first frame and the second frame based at least in part on the first feature vector and the second feature vector to generate clusters of potential object instances.

[0013] Matching may also include, for each cluster in each frame: evaluating the distance between potential object instances within that cluster; and splitting the cluster into multiple clusters based on the evaluation results. Splitting into multiple clusters can be based on the k-means clustering algorithm. The elbow method can be used to determine the parameter k in the algorithm.

[0014] Evaluating the distance between latent object instances in a cluster may include: assigning latent object instances in a cluster to two or more subgroups; determining the centroid of each group based on a first / second feature vector; and evaluating the distance between the centroids. Evaluation may include determining whether splitting the clusters would reduce the distance between the centroids of the subgroups. Splitting the clusters may include defining multiple clusters based on the subgroups.

[0015] Matching can include selecting a single object instance from potential object instances in each cluster of each frame.

[0016] Selection can be performed after clustering. Selection can also be performed after splitting.

[0017] Matching can include matching at least one of the single object instances in the first frame with a single object instance in the second frame.

[0018] Matching can include: evaluating the differences between single object instances in each frame; identifying pairs of single object instances with the lowest differences between them; and matching these pairs of single object instances. This can be repeated to match multiple pairs of single object instances between two frames.

[0019] Differences can be evaluated based on the corresponding feature vectors associated with a single object instance.

[0020] Matching may include rejecting potential object instances based on any one or any combination of two or more of the following: an object confidence score that estimates whether a potential object instance is more likely to be an object or part of the background; a mask confidence score that estimates the likelihood that a mask represents an object; and a mask region.

[0021] Potential object instances can be rejected if their object confidence score, mask confidence score, or mask region falls below a predetermined threshold. All three parameters can be evaluated for each potential object instance, and the instance can be rejected if any one of them falls below the corresponding threshold.

[0022] In addition to or instead of rejecting potential object instances, the same criteria discussed above can be used to select single object instances for each cluster. For example, the single object instance with the highest score can be selected based on any one or two or more of the following: object confidence score, mask confidence score, and mask region.

[0023] Mask confidence scores can be based at least in part on the mask of a latent object instance. Alternatively or additionally, they can be based at least in part on features extracted from regions of interest associated with the latent object instance.

[0024] The mask region can be determined by the number of active pixels in the mask.

[0025] Rejection can be performed after clustering. Rejection can be performed before splitting. Rejection can be performed before selection.

[0026] Mask confidence scores can be generated by a machine learning algorithm trained to predict the degree of correspondence between the mask and the ground truth mask. The degree of correspondence can include the cross-union ratio (CUI) of the mask and the ground truth mask.

[0027] The mask and feature vectors can be generated by a first machine learning algorithm. The mask can be generated by one head of the first machine learning algorithm; the feature vectors can be generated by another head of the machine learning algorithm. The mask confidence score and / or object confidence score can also be generated by another head of the first machine learning algorithm. Therefore, as outlined above, the object confidence score for each potential object instance can be based at least in part on the output of the first machine learning algorithm. The bounded box for each object instance can be generated by the first machine learning algorithm. The bounded box for a given object instance can be refined by one head of the first machine learning algorithm, while the corresponding mask is generated by another head of the first machine learning algorithm.

[0028] The method may further include estimating the motion of at least one matching object in the first frame and the second frame between the first frame and the second frame. Estimating the motion of the object may include, for each of a plurality of pixels of the object: estimating a translational motion vector; estimating a non-translational motion vector; and calculating the motion vector of the pixel as the sum of the translational motion vector and the non-translational motion vector.

[0029] Translational motion vectors can be estimated, at least in part, based on the centroid of the mask in the first frame and the centroid of the corresponding matching mask in the second frame. Alternatively, translational motion vectors can be estimated, at least in part, based on the center of the bounded box in the first frame and the center of the corresponding bounded box in the second frame. Non-translational motion vectors can describe rotation and / or perspective transformations.

[0030] The motion estimation of the object optionally includes: generating a coarse estimate of the motion based at least in part on a mask in the first frame and a corresponding matching mask in the second frame; and refining the coarse estimate using a second machine learning algorithm, wherein the second machine learning algorithm takes the first frame, the second frame and the coarse estimate as input, and is trained to predict the motion difference between the coarse motion vector and the ground reality motion vector.

[0031] In some implementations, the coarse estimate may be a coarse estimate of the translational motion vector for each pixel. In this case, the motion difference may represent the non-translational motion vector (and may optionally also represent a refinement of the translational motion vector). In some implementations, the coarse estimate may include a coarse estimate of both the translational and non-translational motion vectors. In this case, the motion difference may represent a refinement of both the translational and non-translational motion vectors. The coarse estimate of the non-translational motion vector may be generated by another head of the first machine learning algorithm. In any case, the motion difference may also represent the motion of the background.

[0032] A second machine learning algorithm can be trained to predict motion differences at multiple resolutions starting from the lowest resolution and to predict motion differences at successively higher resolutions based on motion differences upsampled from previous resolutions.

[0033] According to another aspect, a method is provided for generating a training dataset for training a machine learning algorithm to perform motion estimation, the method comprising: Obtain multiple images of an object; Obtain multiple images of the background; and Generate multiple pairs of composite images, each pair including a first frame and a second frame. The first frame includes a series of objects superimposed on the selected background, positioned in a first location and orientation. The second frame includes a series of objects superimposed on a selected background, positioned in a second location and in a second orientation, wherein at least some of the objects in the second location and second orientation differ from the first location and first orientation. The method also includes generating: The translational ground motion vector describes the difference between the first and second positions; and The non-translational ground motion vector describes the difference between the first orientation and the second direction.

[0034] The method may also include, for some of the paired composite images, using images of the objects directly as obtained; and for the other paired composite images, modifying the images of the objects before overlaying them onto a background. Modifying the images may include applying the appearance of one object to another (also known as texture mapping).

[0035] The method may further include rejecting some of the multiple images of the obtained object before generating multiple pairs of synthetic images. Rejection may optionally include one or more of the following: rejecting images containing more than a first predetermined number of faces; rejecting images containing fewer than a second predetermined number of faces; and rejecting objects containing multiple non-overlapping portions.

[0036] The translation ground motion vector may include a motion vector that satisfies at least one of the following conditions: the horizontal component of the motion vector is at least 20%, optionally at least 50%, or at least 70% of the width of the first frame; and the vertical component of the motion vector is at least 20%, optionally at least 50%, or at least 70% of the height of the first frame.

[0037] The method may also include dividing multiple pairs of images into a training set for training machine learning algorithms and a test set for testing the performance of machine learning algorithms.

[0038] Each first frame can be generated by randomly selecting objects and randomly positioning them in a first position. Alternatively, the difference between the first and second positions can be randomly selected.

[0039] The method may also include rendering at least one of the following: a translational flow field, comprising a flow field derived from a translational ground-based motion vector; and a combined flow field, comprising a flow field derived from a translational ground-based motion vector and a non-translational ground-based motion vector.

[0040] An image processing system is also provided, comprising: The memory is configured to store a set of frames, including at least a first frame and a second frame; and The first segmentation block is configured to segment the first frame to generate multiple first masks, each first mask identifying a pixel belonging to a potential object instance detected in the first frame; The first feature extraction block is configured to extract a first feature vector characterizing the potential object instance from the first frame for each potential object instance detected in the first frame. The second segmentation block is configured to segment the second frame to generate multiple second masks, each second mask identifying a pixel belonging to a potential object instance detected in the second frame; The second feature extraction block is configured to extract a second feature vector characterizing the potential object instance from the second frame for each potential object instance detected in the second frame; and A matching block is configured to match at least in part with a first feature vector, a first mask, a second feature vector, and a second mask, such that at least one of the potential object instances in the first frame matches one of the potential object instances in the second frame.

[0041] In some implementations, the first and second segmentation blocks may be provided by a single block configured to segment the two frames. Similarly, in some implementations, the first and second feature extraction blocks may be provided by a single block configured to extract feature vectors of potential object instances in the two frames. Alternatively, the corresponding first and second blocks may be separate blocks. For example, this can facilitate parallel processing of the first and second frames. The system may also include a motion estimation block configured to estimate the motion of objects matched by the matching block.

[0042] A processing system is also disclosed, configured to perform the methods outlined above or any one of claims 1 to 8. The processing system may be a graphics processing system or an artificial intelligence accelerator system. The processing system may be embodied in hardware on an integrated circuit.

[0043] A method for manufacturing a processing system as outlined above or as described in any one of claims 9 to 10 is also disclosed using an integrated circuit manufacturing system.

[0044] A method for manufacturing the processing system as outlined above using an integrated circuit manufacturing system is also disclosed, the method comprising: A layout processing system is used to process the computer-readable description of the processing system in order to generate a circuit layout description of the integrated circuits embodying the processing system; and Use integrated circuit manufacturing systems to manufacture processing systems based on circuit layout descriptions.

[0045] Also disclosed is computer-readable code configured to cause the methods outlined above to be executed when the code is run. A computer-readable storage medium on which the computer-readable code is encoded is also disclosed. This computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0046] Also disclosed is a computer-readable storage medium (optionally non-transitory) storing thereon a computer-readable description of the processing system outlined above, which, when processed in an integrated circuit manufacturing system, enables the integrated circuit manufacturing system to: A layout processing system is used to process the computer-readable description of the processing system in order to generate a circuit layout description of the integrated circuits embodying the processing system; and The processing system is manufactured using an integrated circuit fabrication system based on a circuit layout description.

[0047] An integrated circuit manufacturing system is also disclosed, which includes: A non-transitory computer-readable storage medium having stored thereon a computer-readable description of a processing system as outlined above; A layout processing system configured to process a computer-readable description to generate a circuit layout description of an integrated circuit embodying the processing system; and An integrated circuit manufacturing system configured to manufacture a processing system according to a circuit layout description.

[0048] The layout processing system can be configured to determine the location information of the logic components of a circuit derived from a computer-readable description in order to generate a circuit layout description of an integrated circuit that embodies the processing system.

[0049] The processing system can be embodied in hardware on an integrated circuit. A method for manufacturing the processing system at an integrated circuit manufacturing system can be provided. An integrated circuit definition dataset can be provided, which, when processed in the integrated circuit manufacturing system, causes the system to manufacture the processing system. A non-transitory computer-readable storage medium can be provided, on which a computer-readable description of the processing system is stored, which, when processed in the integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the processing system.

[0050] An integrated circuit manufacturing system may be provided, comprising: a non-transitory computer-readable storage medium storing thereon a computer-readable description of a processing system; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the processing system; and an integrated circuit generation system configured to manufacture the processing system according to the circuit layout description.

[0051] Computer program code for performing any of the methods described herein may be provided. A non-transitory computer-readable storage medium may be provided having computer-readable instructions stored thereon, which, when executed at a computer system, cause the computer system to perform any of the methods described herein.

[0052] As will be apparent to those skilled in the art, the above features can be appropriately combined, and can be combined with any aspect of the examples described herein. Attached Figure Description

[0053] The example will now be described in detail with reference to the accompanying drawings, in which: Figure 1 A block diagram of a processing system according to one implementation scheme is shown; Figure 2 This is a flowchart illustrating a part of an object analysis method according to an implementation scheme; Figure 3 This is a flowchart illustrating an object matching method based on an implementation plan; Figure 4 This is a flowchart illustrating a motion estimation method according to one implementation scheme; Figure 5 The illustration shows an instance segmentation method using a neural network based on one implementation scheme. Figure 6 The illustration shows a motion estimation method using neural networks based on one implementation scheme. Figure 7 This is a flowchart illustrating a method for generating a training dataset according to one implementation scheme; Figure 8A computer system is shown in which a processing system according to one embodiment is implemented; and Figure 9 This illustrates an integrated circuit manufacturing system for generating integrated circuits that embody a graphics processing system.

[0054] The accompanying drawings illustrate various examples. Those skilled in the art will understand that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the drawings represent one example of a boundary. In some examples, it may be that one element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the drawings to indicate similar features. Detailed Implementation

[0055] The following description is given by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art.

[0056] The implementation scheme will now be described using examples only.

[0057] definition: As used in this article, a "mask" refers to a pixel map that indicates the location of an object. Masks can be binary. In a binary mask, each pixel marked as active indicates the presence of an object at that pixel. Each pixel marked as inactive indicates the absence of an object at that pixel.

[0058] As used in this article, the centroid of a mask refers to the average position of the active pixels in the mask.

[0059] The inventors have recognized that current state-of-the-art DNN-based motion estimation methods still struggle with challenging situations where objects undergo significant changes in their appearance. They also typically struggle with small, fast-moving objects that move long distances between two frames. In these cases, the object's translation within the frame's field of view may fall outside the limited spatial limits of DNN-based methods. However, due to the human ability to apply a higher level of understanding of objects in a scene, they are often still able to comprehend the motion occurring in such examples. The inventors have further recognized that these challenging situations can be addressed by identifying objects and matching them between two frames.

[0060] While pyramid-based techniques such as PWC-Net extract representations for each square block of the input image at multiple levels, they do not generate object-level representations for each object. Object-level representations can be more expressive when objects undergo complex motion or (non-rigid) objects undergo severe deformation.

[0061] Figure 1 This is a block diagram illustrating an image processing system 100 configured to analyze objects and estimate their motion according to one embodiment. The system includes a memory 110 configured to store a set of frames, including at least a first frame and a second frame. These two frames are also referred to herein as a reference frame and a target frame. The system also includes a segmentation block 120 configured to segment the frames and generate multiple masks for each frame. Each mask identifies the location of a potential object instance in one of these frames. In this context, an object instance is the occurrence of an object in a particular frame. A “potential” object instance is a possible object instance that may be rejected or not rejected at a later time according to predetermined criteria. System 100 also includes: a feature extraction block 130 configured to extract a feature vector for each potential object instance; a matching block 140 configured to match potential object instances between frames; and a motion estimation block 150 configured to estimate the motion of the matched objects.

[0062] Figure 2 This is a flowchart illustrating an object analysis method according to one embodiment. The input to the method includes a first frame and a second frame stored in memory 110. In steps 210 and 220, segmentation block 120 segments the first frame to generate multiple first masks. This is done in two parts. In step 210, segmentation block 120 identifies regions of interest (ROIs) in the first frame. Each ROI is defined by a bounded box. In step 220, for each ROI, segmentation block 120 segments the pixels in the corresponding bounded box to generate a corresponding first mask. In this example, a Region Proposal Network (RPN) is used to identify the ROIs; the RPN is a neural network trained to identify potential object instances. The RPN used in Mask R-CNN (“Mask R-CNN” by K. He, G. Gkioxari, P. Dollár, and R. Girshick, Proceedings of the 2017 IEEE International Conference on Computer Vision, pp. 2961–2969) has been found to produce good results, but other region proposal networks can be used alternatively. The masks are generated by a portion of the same neural network. See below for reference. Figure 5 The segmentation process is described in detail. Steps 210 and 220 are repeated for the second frame by segmentation block 120, which is segmented to generate multiple second masks.

[0063] In this way, segmentation block 120 performs instance segmentation for the first and second frames. Instance segmentation is the problem of identifying instances of objects within an image and providing segmentation for each object. Instance segmentation can be distinguished from object detection, which does nothing more than identify bounded boxes around each object instance. Instance segmentation can also be distinguished from semantic segmentation, in which each pixel is assigned to either the object or background class, but different instances of the object class are not separated from each other. Without being bound by theory, the use of instance segmentation is believed to be at least partially responsible for the good performance of this method.

[0064] In step 230, for each potential object instance identified in the first frame, feature extraction block 130 extracts a first feature vector characterizing that potential object instance from the first frame. Typically, the feature vector is an abstract representation of the object instance, either extracted from the frame or—as in this example—from a feature map derived from the frame. In this example, the feature vector is extracted by another head of the same neural network that identifies bounding boxes and generates masks. In other words, in this example, the neural networks performing these tasks share a common (“backbone”) portion. Feature extraction block 130 also performs the same feature extraction process 230 for the second frame.

[0065] Other tasks are performed by other heads of the neural network. In this specification, these tasks are assigned to feature extraction block 130; however, they can alternatively be assigned to some other block. In step 240, feature extraction block 130 generates an object confidence score that predicts whether a corresponding latent object instance is likely to represent an object or background. In the Mask R-CNN algorithm adapted for use in this example, this object confidence score is referred to as an “objectivity” score. See also: Faster R-CNN (Advances in Neural Information Processing Systems, 2015, pp. 91-99, S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks”).

[0066] In step 250, feature extraction block 130 generates a mask confidence score that predicts whether the corresponding mask is likely to represent an object. Again, in this example, this is generated by another head of the neural network. Specifically, the head of the neural network that generates the mask confidence score is trained to predict the intersection-over-union (IoU) of the mask generated by the mask generation head with its corresponding ground truth mask. This provides an estimate of the accuracy of the generated mask during inference. In this example, this head of the network takes (i) the mask itself and (ii) some of the intermediate features used to generate the mask in earlier layers of the network as input. These inputs are processed by other layers of the neural network to generate the mask confidence score.

[0067] In step 260, the feature extraction block 130 calculates a mask region for each first mask and second mask. Specifically, it counts the number of active pixels in the corresponding mask.

[0068] The data slices generated in steps 210-260 can now be used to match object instances in the first frame with object instances in the second frame. Figure 3 The illustration shows a matching method 300 according to one implementation. In step 310, matching block 140 clusters the set of potential object instances from two frames. In this example, the clustering is based on the HDBScan algorithm (RJ Campello, D. Moulavi, A. Zimek and J. Sander, “Hierarchical density estimates for data clustering, visualization, and outlier detection,” ACM Transactions on Knowledge Discovery from Data (TKDD), Vol. 10, No. 1, p. 5, 2015). The input to the clustering algorithm includes the feature vectors extracted by feature extraction block 130 in step 230.

[0069] Following the initial clustering in step 310, the method proceeds to step 320, where low-quality object instances are rejected by removing them from the clusters. This is done because region proposal networks typically generate too many region proposals, and mask generation may generate multiple masks for each proposed region. Therefore, the number of potential object instances included in the initial clusters is often significantly greater than the number of actual objects. According to this embodiment, three criteria are used in step 320 to reject potential object instances: object instances with an object confidence score below a predetermined threshold are removed; object instances with a mask confidence score below a predetermined threshold are removed; and object instances with a mask region below a predetermined threshold are removed. Of course, fewer criteria and / or other criteria can be used in other embodiments.

[0070] In step 330, matching block 140 splits insufficiently coherent distances. In this example, this is done by the following steps: assigning object instances in a cluster to two or more subgroups; determining the centroid of each subgroup in the feature space (i.e., the centroid of the feature vectors of the object instances that make up the subgroup); evaluating the Euclidean distance between the centroids of the subgroups; and splitting the clusters into smaller clusters in this way, which will significantly reduce the Euclidean distance between the centroids of the subgroups without increasing the number of clusters too much. More specifically, the feature vectors in the original clusters are fed into a k-means clustering algorithm (known to those skilled in the art) to find a smaller set of clusters for the data. The number of clusters k is determined using an elbow method (see, for example, RL Thorndike's "Who belongs in the family" in psychometrics in 1953).

[0071] Next, in step 340, matching block 140 selects single object instances from each cluster present in each frame after step 330. Various criteria can be used to do this. In this example, the potential object instance with the largest mask region in the cluster is selected. The result of this step is that there are many single object instances in each frame. Typically, there may be different numbers of objects in two frames—for example, due to objects arriving or leaving between frames, or due to false positives or false negatives in instance segmentation. In step 350, matching block 140 matches single object instances between the two frames. In this example, this is done using a “greedy” algorithm. In other words, the pairwise distance between the feature vectors of the single object instances is evaluated. The pair with the smallest distance is selected as the first match. From the remaining single object instances, the pair with the second smallest distance is selected as the second match, and so on, until all single object instances in at least one of the two frames have been matched. After step 350 is completed, the method has identified a set of pairwise matches between objects in the first frame and objects in the second frame. This provides useful input to motion estimation algorithm 400.

[0072] Figure 4 This is a flowchart illustrating a motion estimation method 400 according to one embodiment. In step 410, motion estimation block 150 generates a coarse motion estimate for each object. In this embodiment, this includes calculating a translational motion vector between the centroid of each mask in the first frame and the centroid of each matching mask in the second frame. In other words, the coarse estimate is based on the motion of the object centroids. Other methods are, of course, possible. For example, the coarse estimate might be based on the centroid of the bounding box instead of the centroid of the mask.

[0073] In step 420, motion estimation block 150 uses a second neural network to refine the coarse estimate. Specifically, the input to the second neural network includes the first and second frames and the coarse estimate. The second neural network is trained to predict the motion difference between the coarse motion vector and the ground-based motion vector. In this way, the second neural network estimates the non-translation portion of each motion vector. During this process, the second neural network can also refine the translation portion of each motion vector. In this embodiment, the second neural network is trained to predict motion differences at multiple resolutions, starting from the lowest resolution and progressing to successively higher resolutions. The coarse motion estimate is used only as input at lower resolutions. At higher resolutions, the neural network estimates the motion vector based on the refined estimate from the lower resolution.

[0074] The algorithms outlined above have been found to produce good performance in practical experiments. Without being bound by theory, the object matching method is believed to allow the algorithm to handle long-distance motion even of small objects. Refinement using neural network-based motion estimation leverages the improved accuracy of these state-of-the-art motion estimation algorithms. In other words, the advantages of both methods complement each other to produce results better than either method alone could achieve. Each method helps compensate for the limitations of the others.

[0075] Many variations of the algorithm described above are possible. For example, in an alternative implementation, the coarse motion vector estimation includes estimations of both non-translational and translational motion vectors. The coarse non-translational motion vectors can be generated by an additional head of the first neural network described above (i.e., the neural network responsible for segmentation and feature extraction). By providing the second neural network with initial estimates not only of the translational components of the motion vectors but also of the non-translational components, this can help remove some of the burden from the second neural network.

[0076] In other variations, some of these steps can be performed in a different order. For example, the step of rejecting potential object instances can be performed before the clustering step or after the step of splitting clusters. Additionally, as already noted above, it can be performed in a different order than... Figure 2The different orders indicated in the diagram produce different data slices during the segmentation and feature extraction stages. Specifically, different data slices can be generated by different heads of the same neural network. Therefore, they can be generated in any order or, in fact, in parallel.

[0077] As mentioned above, it can be shared. Figure 1 Some of the functions of the different blocks shown—especially when different blocks generate data from different heads of the same backbone neural network. Figure 1 The functionality of each box can be implemented by one or more general-purpose processors or one or more graphics processing units (GPUs). In particular, neural network processing is implemented by one or more neural network accelerators (NNAs).

[0078] Now refer to Figure 5 and Figure 6 A method and system for object matching and motion estimation according to one embodiment are described in detail. It should be understood that this embodiment is merely exemplary, and the scope of this disclosure is by no means limited thereto.

[0079] To estimate the motion field D describing the displacement from the reference frame to the target frame, the problem is divided into two parts. First, for translational motion only, which includes the translational components of the motion of only each object in the scene. First, make a rough estimate. Second, complete and refine the details. This refers to the background motion. and the rotational motion of each object Make an estimate. More specifically, at each object pixel, It is equal to the displacement between frames of the center of the bounded box surrounding the object at that pixel. It is the 2D displacement from the reference frame to the target frame relative to the center of the bounding box in each image. Note that... It is constant within each object in the image, however Within each object, variations and descriptions of both object rotation and perspective projection are presented. The refinement and completion of the estimated motion field in Part Two are achieved through the subsequent network G, thus providing the final flow. .

[0080] It was found by training a modified version of Mask R-CNN. This was in the first frame. I 1 and the second frame I 2. Identifying (potential) object instances Where N is unknown, and for each object instance, a bounded box is computed. bb j , M j Object-oriented scores Oj Mask confidence score S j and representative eigenvectors F j The system matches objects between two frames based on feature vectors, masks, and scores. Then, it generates a sequence using the offset between the mask in the reference frame and the corresponding mask in the target frame. .

[0081] Figure 5 The illustration shows an instance segmentation method based on one implementation scheme. This implementation scheme uses a modified version of the Mask R-CNN algorithm. Figure 5 In the diagram, the box arrow labeled "Conv" represents a convolutional layer of the neural network. The box arrow labeled "FC" represents a fully connected layer. The first and second frames are input to a set of layers represented as the backbone 510 of the neural network. In this example, the backbone 510 consists of a Region Proposal Network (RPN) component of Mask R-CNN (this implementation uses a version of ResNet50 FPN) ("Mask r-cnn" by K. He, G. Gkioxari, P. Dollár, and R. Girshick, pp. 2961–2969, Proceedings of the 2017 IEEE International Conference on Computer Vision). The backbone 510 of the network uses a Feature Pyramid Network (FPN), which generates pyramids of feature maps with different spatial resolutions from fine to coarse. The backbone 510 (RPN) also outputs a coarse estimate of a bounding box for each potential object instance. Further details about FPN can be found in Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, “Feature Pyramid Networks for Object Detection”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2117-2125.

[0082] The feature maps extracted by the backbone 510 of the network are input to two heads of the network with different functions. The first head, consisting of the ROIAlign process 521 and fully connected layers 522, 524, 526, and 528, generates the feature vector and refined bounding box for each latent object instance. The second head, consisting of the ROIAlign process 531, convolutional layers 532, 534, 536, 542, and 544, a max-pooling layer 538, a cascade operation 540, and fully connected layers 546, 548, and 549, generates the mask and mask confidence score for each latent object instance. Each ROIAlign process 521, 531 receives the following as input: (i) the bounding box for each latent object instance; and (ii) the pyramid of feature maps for the entire frame. Each ROIAlign process 521, 531 uses the bounding box for each latent object instance to extract a fixed-size small feature map from the pyramid of feature maps for the entire frame. In this example, for each latent object instance, an appropriate pyramid level is selected based on the size of the bounding box. Bilinear interpolation is then used to extract a fixed-size feature map of the latent object instance from the feature map at the selected level. More specifically: the bounding box is used to clip a portion of the feature map at the selected pyramid level. Bilinear interpolation is then used to pool the clipped feature map into a fixed-size feature map.

[0083] The first head's ROIAlign process 521 takes the roughly estimated bounded boxes generated by the backbone RPN 510 as its bounded box input. The second head's ROIAlign process 531 takes the refined bounded box coordinates generated by the first head as its bounded box input. Table 1 shows the data dimensions under each layer of the network in the current implementation.

[0084] The ROIAlign operation used in Mask R-CNN (“Mask r-cnn” by K. He, G. Gkioxari, P. Dollár, and R. Girshick, pp. 2961–2969, IEEE International Conference on Computer Vision 2017) has been found to produce good results, but alternative alignment mechanisms can be used. In this example, two feature maps are extracted from the backbone 510 of the network. A 7x7x256 feature map is used for the network head that generates feature vectors and (thinned) bounding boxes. A 14x14x256 feature map is used for the network head that generates masks and mask confidence scores. In other examples, feature maps of different sizes can be used.

[0085] Table 1. Data Dimensions of Each Layer in the Modified Mask R-CNN Network

[0086] The modified Mask R-CNN network M typically generates multiple masks and corresponding feature vectors for each instance. Based on these, a single optimal mask and feature vector for each object instance must be selected. The selection is based on a combination of the outputs of M. This implementation uses a combination of object-specific scores and mask confidence scores. The feature vectors corresponding to the selected masks are used to find matches between segmented objects during the object matching process.

[0087] For the set of real objects in the input frame pair Mask R-CNN is typically used for For each object i in the frame, find the set of candidate object instances in frames I1 and I2 respectively. J 1 and J 2. Instead of outputting a class prediction vector, the proposed modified instance segmentation network outputs a feature vector, for example, a 256-dimensional feature vector, for each predicted instance j. Ideally, for two given object instances, if they correspond to the same physical object, the network learns to output feature vectors that are as close as possible in the feature space. If the two instances correspond to different physical objects, the network learns to output feature vectors that are as far apart as possible in the feature space. These feature vectors can be used to identify matching objects in two frames.

[0088] This is achieved by defining the loss function as follows: To achieve this, minimize the process. Let This is the collection of all objects in the frame pair. For each object... Select random feature vectors To represent it. Then select two random subsets. The first set Other matching feature vectors (i.e., those of the same object) The second set is composed of other corresponding feature vectors. By and Exclusion The subsets consist of non-matching feature vectors corresponding to objects other than the specified ones. Both subsets contain as many candidates as possible, up to a maximum of nine.

[0089] Based on these subsets, the similarity loss is then calculated. and dissimilar loss These are defined as follows:

[0090] In both of the above cases, when searching for distance The L1 norm is used. The value w controls the margin for positive and negative dissimilarity and is set to 2 in this example.

[0091] The total loss function for the entire frame pair is then:

[0092] The mask confirmation branch consists of two convolutional layers 542 and 544, which are terminated by three fully connected layers 546, 548, and 549 that receive input from the cascaded operation 540 and compute a mask confidence score for each estimated mask. During training, the loss function minimized is the ratio between this mask confidence score and the integration ratio (IoU) between the estimated mask and the ground truth mask. L 1-norm. In other words, this branch produces an estimate of how accurate the estimated mask is.

[0093] The modified Mask R-CNN searches for numerous candidates across two frames. The proposed method seeks to identify the best candidate corresponding to each object instance in each frame and finds matches between them. This uses the above reference. Figure 3 This will be achieved through the five-stage process already outlined: 1) Initial clustering 2) Remove low-quality candidates 3) Split incorrectly added clusters 4) Select the best candidate 5) Ultimately, greedy matching First (see Figure 3 Step 310) applies a clustering algorithm such as HDBScan. Clustering is performed based on the 256-dimensional feature vector extracted for each object instance during the instance segmentation phase. Then (see...) Figure 3 (Step 320) Remove low-quality instances whose objectivity, mask confidence score, or mask region falls below a threshold. It is still possible for two similar-looking objects from a single frame to form part of the same cluster. To address these issues, if the centroids of instances are too far apart, the k-means elbow method is used to split the clusters from each frame (see [link to step 320]). Figure 3 Step 330). Details of the elbow method can be found in RL Thorndike's "Who Belongs in the Family" in 1953 psychometrics. In step four (see... Figure 3 In step 340), a single best object instance is identified for each cluster in each frame. The selected object instance is the one with the largest mask region. In step five (see...), Figure 3 In step 350), the best instances are greedily matched between the two frames based on the Euclidean distance between the 256-dimensional feature vectors of the two frames. For each estimated mask in the matched objects, and the displacement between the corresponding centroids of those masks, is used to generate a translation-only flow field. .

[0094] Figure 6 The diagram illustrates a method for refining motion estimation of a translation-only flow field. In this implementation, a fitted PWC-Net architecture is used to refine network G; however, other motion estimation networks can be used for this purpose. PWC-Net is a pyramidal network that first estimates the optical flow at low resolution, then successively upsamples and estimates at higher resolutions until the full-scale flow field is obtained. The architecture is modified as follows: Figure 6 As described, incorporating at the minimum three resolutions At the lowest resolution, It is added to the estimated motion field before being upsampled for use in the next resolution. It was resized and rescaled to match the estimate in 6th resolution. In the fifth and fourth scales, a rough estimate is made. and The estimated motion was added to the network in the same way to mimic the residual layers. All three additive terms were weighted by a separate parameter alpha to control the motion independently. The contribution at each resolution. value (in) Figure 6 (Not shown in the image) is trained together with other network weights to optimize the contribution of the translation flow at each level. The inventors' experiments show that... Adding scales beyond the fourth scale does not provide a significant improvement in the results. Without being bound by theory, this is believed to be due to errors in object masks, etc. It contains minor inaccuracies and lacks fine details. If added at the highest resolution... If not, the network will have almost no chance of solving these problems.

[0095] Now will be described in detail Figure 6 The process involves extracting motion estimation features through a series of convolutional layers—layers 611-1 to 611-6 extract features from the first frame, while layers 612-1 to 612-6 extract features from the second frame. The extracted features are then used for multi-resolution motion estimation. At the lowest (6th) resolution, correlation operations 622-6 are used to process the features extracted from the first image. F 6 ( I 1) Features extracted from the second image F 6 ( I 2) Comparison. Features extracted from the first image. F 6 ( I1) The output of the correlation operation 622-6, along with the output of the correlation operation 622-6, is fed as input to the optical flow estimation process 624-6. This process uses a trained neural network to estimate the non-translational portion of the motion vector field. As explained above, a coarse estimate of the translational portion of the motion vector field is provided. The output of optical flow estimation 624-6 is added to produce a combined estimate for the 6th resolution. This estimate is upsampled via upsampling steps 626-6 for use at the next higher resolution (i.e., the 5th resolution). The motion field estimate is then obtained using the upsampled data from the 6th resolution. In step 620-5, the features extracted from the second image are... F 5 ( I 2) Deformation. Assuming the estimation from the 6th resolution is accurate, this will often align features from the first and second images at the 5th resolution. Deformation features from the second image and features from the first image... F 5 ( I 1) Input is fed into correlation operation 622-5. From here, the process continues as for the 6th resolution, calculating correlation 622-5, estimating optical flow 624-5, and adding a coarse estimate of the translation component of the motion vector field. and the combined estimate obtained by upsampling 626-5 The same process is performed at the fourth resolution, with similar reference numerals used for similar operations. Note that, for simplicity, the third, second, and first resolutions are not shown in the figures. This is except for a rough estimate of the translation portion of the motion vector field. In addition to the output of the optical flow estimation, these are processed similarly to those at 6th, 5th, and 4th resolutions. Also note that... Figure 6 The weighting of the trained values ​​by alpha is not shown—again, for simplicity. Details of the construction of the neural network used for optical flow estimation (624) can be found in “Pwc-net: CNNs for optical flow using pyramid, warping, and cost volume” by D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, Proceedings of the 2018 IEEE International Conference on Computer Vision and Pattern Recognition, pp. 8934–8943. It should be understood that this method differs from PWCNet because the neural network is trained to estimate motion residuals at the lowest possible scale, rather than directly estimating the motion field. However, this does not necessarily affect the network's structure.

[0096] Figure 7This is a flowchart illustrating a method for generating a training dataset according to one embodiment. As discussed above, the inventors have found it advantageous to generate motion vector estimates in two parts—namely, translational and non-translational components. This is based on the observation that different algorithms may be better suited to estimating these two different parts of the motion vector field. It should be noted that in the above example, the generation of both the translational and non-translational motion vector components relies on machine learning algorithms—specifically, the use of neural networks. To train such algorithms, suitable training data and relevant ground-based motion vectors are required. It is desirable to divide the ground-based motion vectors into translational and non-translational components to support the training of different parts of the overall motion estimation algorithm. It is difficult or impossible to obtain accurate ground-based motion vectors for any natural frame. Therefore, the inventors propose to synthesize a training dataset. Figure 7 This is an example of a method used to do this. Figure 7 The method can be implemented by any suitable device, including, for example, a suitably programmed computer system. For example, the method can be implemented by... Figure 8 The CPU 902 and / or GPU 904 of the computer system shown in the figure are executing.

[0097] Preferably, the dataset is large and diverse enough to deliver robust performance in machine learning algorithms trained with it. The dataset preferably contains objects that have undergone large translations and whose appearance has changed significantly due to large rotations, occlusion, or lighting variations.

[0098] In step 710, the computer system obtains multiple images of the objects. For this example, the inventors downloaded a large number of 3D object files from Archive3D (https: / / archive3d.net / ). These files were automatically processed to remove any problematic models—for example, models containing too many or too few faces were rejected, and models consisting of multiple disjoint parts were rejected. The remaining models were randomly divided into disjoint “test” and “training” sets of 4,000 and 12,000 objects, respectively. In step 720, the computer system obtains multiple images of the background. In this example, the background is composed of planes textured with images randomly selected from the OpenImages dataset (A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig and V. Ferrari, “The open images dataset v4: Unified imageclassification, object detection, and visual relationship detection atscale,” arXiv:1811.00982, 2018). Random rotations and translations are then applied to the background.

[0099] In step 730, the computer system generates multiple pairs of composite images, each pair comprising a first frame and a second frame. Each first frame includes a set of objects superimposed on a selected background in a first position and a first orientation. The second frame includes the same selection of objects superimposed on the selected background in a second position and a second orientation, wherein at least some of the second positions and second orientations differ from the first positions and first orientations. In this example, this is done by rendering the objects on the background image as described above. When rendering, the objects are either preserved in their initial appearance or randomly textured using natural images from the OpenImages dataset, images with similar textures from ImageAfter (http: / / www.imageafter.com / ), or textures generated procedurally using Blender (BO Community, Blender - a 3D modelling and rendering package, Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018, http: / / www.blender.org). Each image pair in the dataset is generated by selecting a set of random objects, generating textures as outlined above, and generating random trajectories for each object. Of course, image pairs can also be composited in other ways, without necessarily using these source datasets or tools.

[0100] In step 740, the computer system generates a translational ground-based motion vector describing the difference between the first and second positions. The computer system also generates (in step 750) a non-translational ground-based motion vector describing the difference between the first and second orientations. Because the image pairs are synthetic images rather than natural images, ground-based data is always available as part of the synthesis process.

[0101] Note that translated ground motion vectors or non-translated ground motion vectors can be expressed and stored explicitly or implicitly. For example, a computer system can store translated ground motion vectors as well as combined ground motion vectors that are the sum of translated and non-translated components. Those skilled in the art will appreciate that storing one of these components and the sum of two components is equivalent to storing both components.

[0102] In this example, a color image, an object index map, and a ground-based flow plot (combined motion vectors) are rendered. Additionally, the ground-based flow field for the translational components of the motion is generated. These contain the flow fields obtained when each object experiences only the translational component of its trajectory. This additional ground-based flow field can be used to evaluate the translational flow estimation. The accuracy is high. The rendering itself uses the Cycles renderer (https: / / www.cycles-renderer.org / ), which can simulate realistic global illumination (GI) effects.

[0103] All the examples above focus on motion estimation between a pair of frames. Needless to say, the disclosed techniques can be extended to a larger number of frames—for example, by applying them to consecutive pairs of frames in a longer sequence.

[0104] Figure 8 A computer system is shown in which the graphics processing system (including image processing system 100) described herein can be implemented. The computer system includes a CPU 902, a GPU 904, memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 919. A processing block 910 is implemented on the GPU 904 (and...). Figure 1 (Corresponding to processing blocks 120-150 in the original text). In other examples, processing block 910 may be implemented on CPU 902 or on a neural network accelerator (NNA) that may be provided as a supplement or alternative to CPU or GPU. Components of the computer system may communicate with each other via communication bus 920. Temporary register 912 (corresponding to memory 110) is implemented as part of memory 906.

[0105] Figure 1 and Figure 5-6 The data processing system is shown as comprising multiple functional blocks. This is merely illustrative and not intended to define a strict division between different logical elements of such an entity. Each functional block can be provided in any suitable manner. It should be understood that the intermediate values ​​described herein as being formed by the processing system do not need to be physically generated by the processing system at any point in time, and may simply represent logical values ​​that conveniently describe the processing performed by the processing system between its inputs and outputs.

[0106] The processing system described herein may be embodied in hardware on an integrated circuit. The processing system described herein may be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above may be implemented in software, firmware, hardware (e.g., a fixed logic circuit system), or any combination thereof. The terms “module,” “function,” “component,” “element,” “cell,” “block,” and “logic” may be used herein to generally denote software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, cell, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein may be executed by one or more processors that execute code that causes the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that may use magnetic, optical, and other techniques to store instructions or other data and that may be accessible by a machine.

[0107] As used herein, the terms computer program code and computer-readable instructions refer to any kind of executable code for a processor, comprising code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code defining integrated circuits (e.g., hardware description languages ​​or netlists), and code expressed in programming languages ​​such as C, Java, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes the processor of a computer system that supports the executable code to perform tasks specified by said code.

[0108] A processor, computer, or computer system can be any kind of device, machine, or special-purpose circuit, or a collection or part thereof, that has the processing power to execute instructions. A processor can be any kind of general-purpose or special-purpose processor, such as a CPU, GPU, NNA, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0109] This invention also intends to cover software defining the configuration of hardware as described herein, such as hardware description language (HDL) software, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium on which computer-readable program code in the form of an integrated circuit definition dataset is encoded may be provided, which, when processed (i.e., executed) in an integrated circuit manufacturing system, configures the system to manufacture a processing system configured to perform any of the methods described herein, or a processing system including any of the devices described herein. The integrated circuit definition dataset may, for example, be an integrated circuit description.

[0110] Therefore, a method for manufacturing a processing system as described herein can be provided at an integrated circuit manufacturing system. Furthermore, an integrated circuit definition dataset can be provided, which, when processed in the integrated circuit manufacturing system, enables the method for manufacturing the processing system to be executed.

[0111] Integrated circuit definition datasets can be in the form of computer code, such as netlists, code for configuring programmable chips, or hardware description languages ​​suitable for manufacturing at any level in integrated circuits, including register-transfer level (RTL) code, high-level circuit representations (such as Verilog or VHDL), and low-level circuit representations (such as OASIS (RTM) and GDSII). Higher-level representations (such as RTL) that logically define hardware suitable for manufacturing in integrated circuits can be processed on a computer system configured to generate manufacturing definitions of integrated circuits within the context of a software environment that includes definitions of circuit elements and rules for combining these elements to generate the manufacturing definition of the integrated circuit defined by that representation. As is typically the case where software executes at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate the manufacturing definition of the integrated circuit, executing code that defines the integrated circuit in order to generate the manufacturing definition of the integrated circuit.

[0112] Now refer to Figure 9 This describes an example of processing integrated circuit definition datasets at an integrated circuit manufacturing system in order to configure the system as a manufacturing processing system.

[0113] Figure 9An example of an integrated circuit (IC) manufacturing system 1002 is shown, which is configured to manufacture a processing system as described in any of the examples herein. Specifically, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g., defining a processing system as described in any of the examples herein), process the IC definition dataset, and generate an IC (e.g., embodying the processing system as described in any of the examples herein) based on the IC definition dataset. The processing of the IC definition dataset configures the IC manufacturing system 1002 to manufacture integrated circuits embodying the processing system as described in any of the examples herein.

[0114] The layout processing system 1004 is configured to receive and process an IC definition dataset to determine a circuit layout. Methods for determining a circuit layout based on an IC definition dataset are known in the art and may involve, for example, synthesizing RTL code to determine the gate-level representation of the circuit to be generated, for example, in relation to logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined based on the gate-level representation of the circuit. This can be done automatically or with user intervention to optimize the circuit layout. Once the layout processing system 1004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0115] As is known in the art, IC generation system 1006 generates ICs according to a circuit layout definition. For example, IC generation system 1006 can implement a semiconductor device manufacturing process for generating ICs, which may involve a multi-step sequence of photolithography and chemical processing steps, during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask, which can be used in the photolithography process to generate ICs according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of computer-readable code, which IC generation system 1006 can use to form a suitable mask for generating ICs.

[0116] The various processes performed by the IC manufacturing system 1002 can all be implemented in one location, for example, by one party. Alternatively, the IC manufacturing system 1002 can be a distributed system, allowing some processes to be performed in different locations and by different parties. For example, some of the following stages can be performed in different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition dataset to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0117] In other examples, processing of an integrated circuit definition dataset at an integrated circuit manufacturing system can configure the system to manufacture a processing system without processing the IC definition dataset to determine circuit layout. For example, an integrated circuit definition dataset can define the configuration of a reconfigurable processor, such as an FPGA, and processing of the dataset can configure the IC manufacturing system (e.g., by loading configuration data into the FPGA) to generate a reconfigurable processor with the defined configuration.

[0118] In some implementations, when processed in an integrated circuit manufacturing system, the integrated circuit manufacturing definition dataset can enable the integrated circuit manufacturing system to generate devices as described herein. For example, the integrated circuit manufacturing definition dataset referenced above... Figure 9 The described method configures an integrated circuit manufacturing system to manufacture devices as described herein.

[0119] In some examples, an integrated circuit definition dataset may include software running on hardware defined at the dataset, or software running in combination with hardware defined at the dataset. Figure 9 In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined at the integrated circuit definition dataset during the manufacturing of the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0120] Compared to known implementations, the implementation of the concepts set forth in this application in devices, apparatuses, modules, and / or systems (and in the methods implemented herein) can lead to performance improvements. Performance improvements may include one or more of increased computational performance, reduced latency, increased throughput, and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules, and systems (e.g., in integrated circuits), trade-offs can be made between performance improvements and physical implementation methods, thereby improving manufacturing methods. For example, a trade-off can be made between performance improvements and layout area, thereby matching the performance of known implementations but using less silicon. This can be accomplished, for example, by reusing functional blocks serially or sharing functional blocks among elements of a device, apparatus, module, and / or system. Conversely, the concepts set forth in this application that lead to improvements in the physical implementation of devices, apparatuses, modules, and systems (such as reduced silicon area) can be traded off for performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0121] The applicant has independently disclosed each individual feature described herein, as well as any combination of two or more such features, to the extent that such features or combinations can be implemented based on the specification as a whole, in accordance with the common knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein. In view of the foregoing description, those skilled in the art will understand that various modifications can be made within the scope of this invention.

Claims

1. A method for analyzing one or more objects in a set of frames including at least a first frame and a second frame, the method comprising: The first frame is segmented to generate a plurality of first masks, each first mask identifying a pixel belonging to a potential object instance detected in the first frame; For each potential object instance detected in the first frame, a first feature vector characterizing the potential object instance is extracted from the first frame; The second frame is segmented to generate multiple second masks, each second mask identifying a pixel belonging to a potential object instance detected in the second frame; For each potential object instance detected in the second frame, a second feature vector characterizing the potential object instance is extracted from the second frame; as well as At least in part based on the first feature vector, the first mask, the second feature vector, and the second mask, at least one of the potential object instances in the first frame matches one of the potential object instances in the second frame. Its features are, The matching includes clustering the potential object instances detected in the first frame and the second frame based at least in part on the first feature vector and the second feature vector to generate clusters of potential object instances; Select a single object instance from the potential object instances in each cluster of each frame; At least one of the single object instances in the first frame is matched with a single object instance in the second frame.

2. The method of claim 1, wherein the matching further comprises, for each cluster in each frame: Evaluate the distance between the potential object instances in the cluster within the frame; and Based on the evaluation results, the cluster is split into multiple clusters.

3. The method of claim 1, wherein the matching comprises rejecting potential object instances based on any one or more of the following, or any combination thereof: The object confidence score estimates whether a potential object instance is likely to be an object or part of the background. Mask confidence score, which estimates the probability that the mask represents an object; and Mask area, The mask confidence score is optionally generated by a machine learning algorithm trained to predict the degree of correspondence between the mask and the ground reality mask.

4. The method of claim 1, wherein the first mask and the second mask, the first feature vector and the second feature vector are generated by a first machine learning algorithm.

5. The method of claim 1, further comprising, for at least one matching object in the first frame and the second frame, estimating the motion of the object between the first frame and the second frame. Optionally, estimating the motion of the object includes, for each of a plurality of pixels of the object: Estimate the translational motion vector; Estimate the non-translational motion vector; and The motion vector of the pixel is calculated as the sum of the translational motion vector and the non-translational motion vector.

6. The method of claim 5, wherein estimating the motion of the object comprises: A rough estimate of the motion is generated, at least in part, based on the mask in the first frame and the corresponding matching mask in the second frame; as well as A second machine learning algorithm is used to refine the coarse estimate. The second machine learning algorithm takes the first frame, the second frame, and the coarse estimate as input, and The second machine learning algorithm was trained to predict the motion difference between the approximate motion vector and the ground-based real-world motion vector. Optionally, the machine learning algorithm is trained to predict the motion differences at multiple resolutions starting from the lowest resolution and to predict the motion differences at successively higher resolutions based on upsampling of the motion differences from previous resolutions.

7. An image processing system, comprising: A memory configured to store a set of frames including at least a first frame and a second frame; and A first segmentation block is configured to segment the first frame to generate a plurality of first masks, each first mask identifying a pixel belonging to a potential object instance detected in the first frame; A first feature extraction block is configured to extract a first feature vector characterizing the potential object instance from the first frame for each potential object instance detected in the first frame; A second segmentation block is configured to segment the second frame to generate a plurality of second masks, each second mask identifying a pixel belonging to a potential object instance detected in the second frame; The second feature extraction block is configured to extract a second feature vector characterizing the potential object instance from the second frame for each potential object instance detected in the second frame; and A matching block, configured to match at least in part with the first feature vector, the first mask, the second feature vector, and the second mask, such that at least one of the potential object instances in the first frame matches one of the potential object instances in the second frame. Its features are, In order to perform the matching, the matching block is configured as follows: Clustering of the potential object instances detected in the first frame and the second frame is performed at least in part based on the first feature vector and the second feature vector to generate a cluster of potential object instances; Select a single object instance from the potential object instances in each cluster of each frame; At least one of the single object instances in the first frame is matched with a single object instance in the second frame.

8. The image processing system of claim 7, wherein the first segmentation block and the second segmentation block are the same segmentation block, and / or the first feature extraction block and the second feature extraction block are the same feature extraction block.

9. A method of manufacturing the processing system of claim 7 or 8 using an integrated circuit manufacturing system, the method comprising: A layout processing system is used to process the computer-readable description of the processing system in order to generate a circuit layout description of the integrated circuits representing the processing system. as well as The processing system is manufactured using an integrated circuit manufacturing system based on the circuit layout description.

10. A computer-readable storage medium storing computer-readable code configured to perform the method of any one of claims 1 to 6 when the code is executed.

11. A computer-readable storage medium storing thereon a computer-readable description of a processing system as claimed in claim 7 or 8, wherein, when processed in an integrated circuit manufacturing system, the computer-readable description causes the integrated circuit manufacturing system to: A layout processing system is used to process the computer-readable description of the processing system to generate a circuit layout description of the integrated circuits embodying the processing system; and The processing system is manufactured using an integrated circuit manufacturing system based on the circuit layout description.

12. An integrated circuit manufacturing system, comprising: A computer-readable storage medium having a computer-readable description thereon of the processing system as described in claim 7 or 8; A layout processing system configured to process the computer-readable description in order to generate a circuit layout description of an integrated circuit embodying the processing system; and An integrated circuit manufacturing system configured to manufacture the processing system according to the circuit layout description.

Images