Robust object recognition using object geometry templates

Abstract

A system and computer-implemented method are provided for performing object recognition based on a geometric template of an object (210) to be identified and based on input data, which represents input points (200) and is obtained in some embodiments by extracting points of interest from sensor data. Therefore, the object recognition operates in a geometric domain that can represent an abstraction of the sensor data, thus relying less on a specific type of sensor data or a specific type of sensor, whereas otherwise it would be a case of machine learning-based object recognition learned directly on said sensor data.

Description

Technical Field

[0001] This invention relates to a system and a computer-implemented method for object recognition using a geometric template of an object. The invention also relates to a computer-readable medium comprising data representing instructions arranged to cause a processor system to perform the computer-implemented method. Background Technology

[0002] Object recognition is a key enabling factor in various technological applications, such as autonomous driving, where it can be used to detect and identify objects in sensor data acquired through one or more sensors integrated within the vehicle. For example, other road users (cars, pedestrians, etc.) or traffic signals can be identified from image data acquired by camera sensors. In some cases, the output of object recognition can be used to control actuators, such as those involved in steering and / or braking of the vehicle, to autonomously avoid collisions with other road users, for example.

[0003] Object recognition has a variety of other uses. For example, in robotics, object recognition enables robots to identify objects in their environment and act accordingly. However, object recognition is also used to enable character recognition, pose recognition, searching in images, and more.

[0004] Machine learning techniques are known for object recognition. This typically involves training a learnable model (e.g., a neural network) with a large dataset of training data that analogizes sensor data types to the data types for which the learnable model is to be applied. Disadvantageously, this training data typically requires manual annotation, which can be time-consuming and expensive. Furthermore, if the characteristics of the sensor data change after deployment (e.g., due to a different type of camera used), the machine learnable model may need to be retrained. Summary of the Invention

[0005] The goal is to achieve object recognition that addresses one or more of the shortcomings of the machine learning-based methods described above.

[0006] The following measures address one or more of these disadvantages by performing object recognition in the geometric domain based on a geometric template of the object to be identified and based on input data representing input points, which in some implementations can be obtained by extracting points of interest from sensor data.

[0007] According to a first aspect of the invention, a computer-implemented method and system for object recognition are provided. According to another aspect of the invention, a computer-readable medium is provided, the computer-readable medium comprising data representing instructions arranged to cause a processor system to perform the computer-implemented method.

[0008] The above measures provide computer-implemented methods and systems for object recognition in a geometric domain. Specifically, input data, including input points defined as coordinates in a coordinate system (e.g., 3D or 2D coordinates in 3D Euclidean space), can be accessed. Such points can be obtained, for example, by processing sensor data to extract points of interest. This result, analogous to a cloud in 3D space, might be referred to elsewhere as an "input point cloud." Additionally, template data, comprising a finite number of template points defined as coordinates in a coordinate system of the same type, can be accessed. These template points can represent a geometric template of an object, as each point can represent a point of interest in the object, and the relative positions of these template points can indicate the overall geometry of the object. For example, the template points can define points of interest on the exterior of a vehicle, such as corners. The template points can be labeled to distinguish corresponding parts of the object. Such labeling can, but does not necessarily, represent semantic labeling. In some implementations, the annotation can be a continuous index of template points in the coordinate system, such as 1, 2, 3, 4, etc.

[0009] Object recognition can then be performed in the geometric domain based on the following steps: An undirected weighted graph G = (V, E) can be constructed, having nodes V representing the input points and edges E between the nodes. The occurrence or local presence of objects in the input data can now be detected based on systems and methods that assign edge labels to edges E to define cuts in the graph, thereby decomposing the graph into one or more clusters representing (local) objects. These edge labels can represent binary labels defining connectivity (e.g., 1) or cuts (e.g., 0) between pairs of nodes, thus partitioning the graph into disjoint subsets representing clusters of nodes. This type of node clustering is known in graph theory. Additionally, node labels can be assigned to nodes V to identify the corresponding node as representing a corresponding part of the corresponding object. Therefore, object recognition can identify (local) objects in the input data by forming node clusters in the graph through edge cutting, and labeling the nodes of the clusters based on the labels of all or a subset of template points. If the cluster contains all the labels of the template points, the object may be detected completely in the input data; otherwise, the object may be detected locally.

[0010] Therefore, object recognition can be performed using edge and node annotations, which in turn may involve the following: First, a cost function can be provided that expresses the scale-invariant similarity between a given triangle in the input points and a given triangle in the geometric template. In other words, the cost function determines the similarity between a pair of three input points and three template points in a scale-invariant manner. Here, "scale-invariant" can mean that the similarity measure is invariant to similarity transformations (such as translation, rotation, reflection, and scaling). The similarity measure can be constrained in various ways, but it is generally acceptable to constrain it to have a negative contribution to the cost function, since the maximum similarity may correspond to a local minimum in the cost function. Furthermore, for a pair of corresponding triangles, the similarity measure contributes to the cost function only if the annotation of the node matches the annotation of the template point and the node is part of the same cluster; otherwise, it contributes nothing (i.e., provides zero contribution). Therefore, similarity metrics can represent the conditional components of the cost function and can reward edge and node labelling that creates a cluster of nodes whose labels match the labels of the template points or at least a subset of the template points.

[0011] Based on the cost function, the first local search algorithm can be applied to the initial form of an undirected weighted graph that can be partitioned to the maximum extent, because each edge label can define a cut edge that causes all nodes to cluster in discrete clusters, where the nodes are not yet labeled. The first local search algorithm (which can be a so-called greedy algorithm) can be used to obtain initial edge labels and node labels, which, according to the cost function, can represent local minima. However, these local minima cannot guarantee or may not actually represent the global minimum, but can be regarded as the "initial solution" to the object recognition problem. In determining the initial edge labels and node labels, the first local search algorithm can connect nodes through edge labels during the iteration process to form labeled nodes and clusters in the graph, thereby obtaining the initial solution.

[0012] Having obtained the initial edge and node labels, a second local search algorithm can be applied, which seeks an improvement upon the initial local minimum. Compared to the first local search algorithm, the second local search algorithm may be able to iteratively move nodes between clusters (rather than simply connecting nodes) by re-labeling edges and re-labeling nodes. Therefore, the second local search algorithm can obtain a minimum, which typically represents an improvement relative to the initial solution, or at least the same solution if no improvement is achieved. The result can be a partitioned and labeled graph. The assigned edge and node labels can then be output as object identification data, representing the result of object identification. As will be explained elsewhere, the object identification data can then be used for various purposes, including but not limited to controlling one or more actuators based on the object identification data.

[0013] The above measures enable object recognition based on a geometric template of the object to be identified and on input data, which represents input points and, in some implementations, can be obtained by extracting points of interest from sensor data. Therefore, operating the object recognition in a geometric domain, which can represent an abstraction of the sensor data, reduces reliance on a specific type of sensor data or a specific type of sensor, whereas machine learning-based object recognition might otherwise be a direct learning of sensor data. In particular, by using scale-invariant similarity metrics, object recognition may be less sensitive to various geometric deformations (including perspective projection) between the input data. Furthermore, using triangle-based similarity metrics allows for the detection of local objects, as locally detected objects may also contribute to local minima of the cost function. This local detection is crucial in real-world applications where an object is frequently partially occluded by another object. For example, a car might be partially occluded by a sign; however, detecting the car is still valuable because it may represent an obstacle for an autonomous driving system.

[0014] The above measures further establish a two-step method specifically for object recognition in geometric domains by sequentially applying two different local search algorithms. This is based on the insight that the performance of local search algorithms, which can move nodes between clusters and thus have a large degree of exploratory freedom, can be highly dependent on initialization, as poor initialization (e.g., completely unlabeled and disconnected graphs or randomly labeled edges and nodes) typically yields (very) poor results. Therefore, the local search algorithm is applied to the graph beforehand with the aim of providing only acceptable initial solutions. This "first" local search algorithm iteratively connects and labels nodes, thus having a lower degree of exploratory freedom compared to the second local search algorithm, and thus providing acceptable results even for completely unlabeled and disconnected graphs. After obtaining the initial solution in the form of the initial edge and node labels, the second local search algorithm can then seek improvements by utilizing its greater degree of exploratory freedom to obtain an initial local minimum. Therefore, the two-step method overcomes the initialization problem in graph-based object recognition.

[0015] Optionally, the cost function The similarity between corresponding triangles is expressed as a function constraining the angles of those triangles. It has been found that constraining the angles of the corresponding triangles is well-suited for scale-invariant similarity measures.

[0016] Optionally, the method further includes: determining the similarity as a function defining the angle of the corresponding triangle by projecting the angle of the triangle as 3D coordinates onto a 2D simplex and by calculating the distance between the coordinates of the triangle projected onto the 2D simplex.

[0017] Assuming that angles define triangles and also define 3D coordinates in a 3D coordinate system (e.g., (α, β, γ) represent angles α, β, and γ), the similarity between two triangles can be expressed as the distance between their 3D coordinates. However, since the 3D coordinates in the 3D coordinate system lie on the 2D simplex and the sum of the angles is always 180°, a similarity measure can be obtained by determining the distance on the 2D simplex (e.g., by using the Mahalanobis distance).

[0018] Optionally, the second local search algorithm is the Kernighan-Lin (KL) algorithm. It has been found that, based on a given cost function, the KL algorithm is well-suited for identifying (local) objects in the input data by providing node and edge labels in the graph. However, it has been found that the KL algorithm performs poorly when weakly initialized. Therefore, according to the measures above, executing a first local search algorithm that provides initial node and edge labels before the KL algorithm unexpectedly and significantly improves overall object recognition.

[0019] Optionally, the first local search algorithm is the Greedy Additive Edge Contraction (GAEC) algorithm. It has been found that the GAEC algorithm can provide acceptable initialization for the KL algorithm based on unlabeled graphs.

[0020] Optionally, the geometric template data includes template points that represent a compound geometric template of multiple objects, wherein the template points are labeled to distinguish corresponding parts of corresponding objects among the multiple objects.

[0021] Optionally, the method further includes accessing sensor data acquired by the sensor and generating input points by extracting points of interest from the sensor data. This sensor data can be accessed via a sensor data interface from any type of sensor that provides sensor data from which geometrically defined points of interest can be extracted. This can include not only sensors that capture image data, such as camera sensors, but also depth sensors (time-of-flight, range), lidar sensors, radar sensors, etc.

[0022] Optionally, the sensor data is image data acquired by a camera, and the method includes generating the input points by extracting points of interest from the image data as keypoints. Such keypoints may, for example, represent corners.

[0023] Optionally, the image data is 2D image data, and the method includes generating the input points by extracting the key points from the 2D image data and back-projecting the extracted key points into a 3D coordinate system based on the depth information of the 2D image data. In practice, by combining the 2D image data with depth information (obtained, for example, from a stereo camera system or an additional depth sensor), 3D key points can be reconstructed and used as input for the object recognition.

[0024] Optionally, the template points are labeled by continuously indexing them within the coordinate system. Semantic labeling may not be necessary for the cost function, but simple continuous indexing may be sufficient to determine whether the labeling of the node matches the labeling of the template point.

[0025] Optionally, the system also includes a sensor interface for accessing sensor data acquired by the sensor, wherein the processor subsystem is configured to generate the input point by extracting the point of interest from the sensor data.

[0026] Optionally, the system further includes an actuator interface for providing control data to the actuator, wherein the processor subsystem is configured to generate the control data based on the object identification data to control the actuator.

[0027] Alternatively, a vehicle or robot including the system is provided. For example, the vehicle or robot can use the object recognition to identify objects in its environment.

[0028] Those skilled in the art will understand that two or more of the above embodiments, implementations and / or optional aspects of the invention can be combined in any manner that they deem useful.

[0029] Those skilled in the art can perform any system, any computer-implemented method, or any computer-readable medium (corresponding to the modifications and variations described by another entity in the foregoing) based on this specification. Attached Figure Description

[0030] These and other aspects of the invention will become apparent and further elucidated by referring to the embodiments described below by way of example, and with reference to the accompanying drawings, in which:

[0031] Figure 1 A system for object recognition is shown, which includes a sensor interface for receiving sensor data from sensors in the environment and an actuator interface for controlling actuators to move in the environment.

[0032] Figure 2 The system integrated into an autonomous vehicle is shown;

[0033] Figure 3A The input point cloud representing a star is shown;

[0034] Figure 3B An example of object recognition of constellations is shown, which is applied to the input point to identify stars based on a geometric template of constellations.

[0035] Figure 4 A computer-implemented method for object recognition is shown; and

[0036] Figure 5 A computer-readable medium containing data is shown.

[0037] It should be noted that these figures are schematic only and are not drawn to scale. In the figures, elements corresponding to those already described may have the same reference numerals.

[0038] Reference list of numbers

[0039] The following list of reference numerals is provided to aid in interpreting the drawings and should not be construed as limiting the claims.

[0040] 20 sensors

[0041] 22 cameras

[0042] 40 Actuators

[0043] 42 Electric motor

[0044] 60 Environment

[0045] 80 Autonomous vehicles

[0046] 100 System

[0047] 120 Sensor Interface

[0048] 122 Sensor Data

[0049] 140 Actuator Interface

[0050] 142 Control Data

[0051] 160 Processor Subsystem

[0052] 180 Data Storage Interface

[0053] 190 Data Storage

[0054] 192 Template Data

[0055] 194 Input Data

[0056] 196 Object Recognition Data

[0057] 200 represents the input point cloud of stars.

[0058] 210 represents an object representing a star cluster (the Big Dipper).

[0059] 400 Methods for Object Recognition

[0060] 410 Access Template Data

[0061] 420 Access Input Data

[0062] 430 Cost Function Calculation

[0063] 440 Greedy Additive Edge Contraction (GAEC)

[0064] 450 Kernighan-Lin (KL) Algorithm

[0065] 460 Output clustering and node labels

[0066] 500 computer-readable media

[0067] 510 Non-temporary data Detailed Implementation

[0068] Figure 1 A system 100 for object recognition is shown. System 100 may include an input interface for accessing template data as described elsewhere and for accessing input data for object recognition. For example, also as Figure 1 As illustrated, the input interface may include a data storage interface 180 that can access template data 192 and input data 194 from data storage 190. For example, data storage interface 180 may not only be a memory interface or persistent storage interface (e.g., a hard disk interface or SSD interface), but also a personal network interface, local area network interface, or wide area network interface (e.g., a Bluetooth interface, Zigbee interface, Wi-Fi interface, Ethernet interface, or fiber optic interface). Data storage 190 may not only be internal data storage of system 100, such as a hard disk drive or SSD, but also external data storage, such as network-accessible data storage. In some embodiments, template data 192 and input data 194 may each be accessed from different data storages, for example, via different subsystems of data storage interface 180. Each subsystem may be of the type of data storage interface 180 as described above.

[0069] Template data 192 may include template points, which are defined as coordinates in a coordinate system and represent the geometric template of the object. Template points can be labeled in the template data to distinguish corresponding parts of the object. Input data 194 may include input points, which are defined as coordinates in a coordinate system.

[0070] System 100 may further include processor subsystem 160, which may be configured to construct an undirected weighted graph (the undirected weighted graph having nodes representing input points and edges located between the nodes) during operation of system 100, and perform object recognition by finding one or more occurrences or partial occurrences of objects in the input data based on the following steps: i) assigning edge labels to edges to define cut edges in the graph to obtain decomposition into one or more clusters representing objects, and ii) assigning node labels from a set of labels to nodes to identify the corresponding nodes as representing the corresponding parts of the corresponding objects. More specifically, edge and node annotations (which can be performed by processor subsystem 160) may include: j) providing a cost function that expresses the scale-invariant similarity between a corresponding triangle in a node and a corresponding triangle in a template point, wherein scale-invariant similarity contributes to the cost function only when the annotation of a node matches the annotation of a template point and the nodes are part of the same cluster; jj) applying a first local search algorithm to an initial graph having each unannotated node and each labeled edge to define a cut edge, wherein the first local search algorithm is configured to iteratively connect nodes by edge annotation to form clusters in the graph and to search for an initial local minimum in the cost function by iteratively labeling nodes, thereby obtaining initial edge and node annotations; and jjj) applying a second local search algorithm to the initial edge and node annotations, wherein the second local search algorithm is configured to find an improvement to the initial local minimum of the cost function by relabeling edges and nodes to iteratively move nodes between clusters. As a result, processor subsystem 160 can obtain a graph with labels assigned to nodes and edges.

[0071] System 100 may also include an output interface for outputting object recognition data representing the assigned edge labels and node labels. For example, as well as... Figure 1 As illustrated, the output interface may include a data storage interface 180, which in these embodiments is an input / output (“IO”) interface, via which object identification data 196 may be stored in the data storage 190. Alternatively, the output interface may be separate from the data storage interface 180, but may typically have the type of data storage interface 180 as described above.

[0072] Figure 1Several optional components of system 100 are also shown. For example, in some embodiments, system 100 may include a sensor interface 120 for accessing sensor data 122 acquired by sensor 20 in environment 60. In such embodiments, processor subsystem 160 may be configured to generate input points by extracting points of interest from the sensor data. Thus, in these embodiments, sensor interface 120 may represent an input interface as described above. However, in some embodiments, system 100 may also have a data storage interface 180 for storing the acquired sensor data 122 and / or input data 194 derived by processor subsystem 160 from sensor data in data storage 190. Typically, sensor interface 120 may have any suitable form, including but not limited to low-level communication interfaces such as I2C or SPI-based data communication, or data storage interfaces of the type described above for data storage interface 180.

[0073] In some implementations, system 100 may include an actuator interface 140 for providing control data 142 to actuator 40 in environment 60. This control data 142 may be generated by processor subsystem 160 to control actuator 40 based on object identification data. For example, the actuator may be an electric actuator, hydraulic actuator, pneumatic actuator, thermal actuator, magnetic actuator, and / or mechanical actuator. Specific, but non-limiting, embodiments include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc. Reference Figure 2 This type of control is described for autonomous vehicles.

[0074] In other implementation schemes ( Figure 1 (Not shown in the image), system 100 may include an output interface to a rendering device, such as a display, light source, speaker, vibration motor, etc. The rendering device may be used to generate sensory-perceptible output signals that can be generated based on object recognition data, such as to identify the recognized object to the user, or generally to provide any other type of sensory-perceptible output signal derived, such as for use in guidance, navigation or other types of control.

[0075] Typically, System 100 may be embodied as a single device or apparatus (such as a workstation or server) or embodied within a single device or apparatus. The server may be an embedded server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem may be embodied not only by a single central processing unit (CPU), but also by a combination or system of such a CPU and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory (such as RAM) or non-volatile memory (such as Flash). Alternatively, the system's processor subsystem may be implemented in a device or apparatus in the form of programmable logic (e.g., as a field-programmable gate array (FPGA)). Typically, each functional unit of System 100 may be implemented in the form of a circuit. System 100 may also be implemented in a distributed manner, for example, involving different devices or apparatuses (such as distributed local servers or cloud-based servers). In some embodiments, System 100 may be part of a vehicle, robot, or similar physical entity, and / or may represent a control system configured to control the physical entity.

[0076] Figure 2 One embodiment of the above is shown, in which system 100 is illustrated as a control system for an autonomous vehicle 80 operating in environment 60. The autonomous vehicle 80 can be incorporated into system 100 to control the steering and braking of the autonomous vehicle based on sensor data obtained from a video camera 22 integrated within the vehicle 80. For example, in the event of an anticipated collision between the autonomous vehicle 80 and an obstacle, system 100 can control an electric motor 42 to perform (regenerative) braking. Obstacles can be identified by system 100 in the sensor data by extracting key points from the sensor data and applying object recognition (e.g., in a manner described elsewhere in this specification) to the extracted key points.

[0077] The following examples describe geometric object detection in more detail. However, practical implementations of geometric object detection can be performed in many other ways, such as based on analogous mathematical concepts.

[0078] Object detection using graph-based representations of input points can generally be considered a weighted correlation clustering problem about graphs, which is known in computer science. Before considering measures to solve this problem, let's first consider its formation.

[0079] The high-order weighted correlation clustering problem with node labels can be rigorously formulated as a binary multilinear procedure. A feasible solution to this problem can be constrained by both the clustering of the given graph and the node labeling. An example of a multilinear problem can be constrained based on the following data:

[0080] • The internal clusters of the graph G = (V, E) are correlated.

[0081] A finite non-empty set L is called the node label set.

[0082] • A family of connected subsets of nodes

[0083] For any set in this family and any label λ of this set U :U→L, It is called if and only if all nodes in U are according to λ U The cost to be paid when labeled and assigned to the same cluster.

[0084] Clusters in a graph can be identified by the set of edges that span different clusters. This set of edges is often called a "multicut" of the graph and has the following property: no cycle intersects a multicut edge on exactly one edge. Specifically, consider a set x with edges labeled "0". -1 The binary label x of the edge (0): the set X of E→{0, 1} is used to constrain a multi-cut edge, thus constraining the clustering of the graph. For any edge {v, w}∈E, x vw =0 can indicate that event nodes v and w are in different clusters, or x vw =1 indicates that v and w are in the same cluster. Formally:

[0085]

[0086] We can further consider the function y: V×L→{0, 1}, which, for any node v∈V and any label l∈L, passes through y vl =1 indicates that node v has been assigned the label l, or via y vl =0 indicates that node v has not been assigned a label l. To ensure that each node is assigned an exact label, these functions can be constrained to a set:

[0087]

[0088] With regard to G, L, Instances of the higher-order weighted correlation clustering problem of node labels with c can be constrained to binary multilinear procedures:

[0089]

[0090] Note that if all nodes in set U are assigned to the same cluster, the first product will use the value "1"; otherwise, it will use the value "0". Also note that if according to λ... U If a node in set U is labeled, the second product will use the value "1"; otherwise, it will use the value "0". Therefore, the second product is valid if and only if the nodes in set U are labeled according to λ. U Costs can only be paid when the data is labeled and assigned to the same cluster.

[0091] A feasible solution to equation (3) can be obtained using a local search algorithm. In particular, a combination of a greedy additive edge contraction algorithm and a Kernighan-Lin algorithm can be used, which is summarized as a higher-order cost function and label in the following manner.

[0092] The problem to be solved is object recognition. As input, a geometric template is obtained, which can be derived from... The set of tags is represented by L = {1, ..., l}, which is a finite set of indices. The tag set can be easily constrained by continuously indexing the template points. The goal is to find all occurrences of the template in the set of input points, along with the labels of those points. It is assumed that the template can undergo similarity transformations such as translation, rotation, reflection, and scaling.

[0093] The following defines a model robust to similarity transformation. For this purpose, a ternary cost is defined at three points (corresponding to triangles) such that similar triangles are assigned the same cost, i.e., the cost is scale-invariant. The cost also depends on a label that identifies the corresponding triangle from the template.

[0094] make To project the coordinates of a triangle onto a 2D simplex Functions:

[0095]

[0096] Mahalanobis distance It can be used in simplex Above, among which and These are the mean vector and the covariance matrix, respectively. For all Can be limited

[0097]

[0098] in and It is the covariance matrix of the corresponding triangle from the template. Then, the ternary cost can be constrained as:

[0099]

[0100] Note that by setting the cost between all templates to ∞, so that different labels are assigned to points in different groups, the model can handle multiple templates.

[0101] The following sections will first introduce the pseudocode of 1) the Kernighan-Lin algorithm, 2) the update function used in the Kernighan-Lin algorithm, and 3) the Greedy Additive Edge Contraction (GAEC) algorithm, and then provide comments on the pseudocode.

[0102]

[0103]

[0104] The Kernighan-Lin algorithm was originally proposed for set partitioning problems and has since been adapted for multi-cut edge problems with node labels and higher-order cost functions. The variant of the Kernighan-Lin algorithm described in this specification is specifically adapted for higher-order node-labeled multi-cut edge problems.

[0105] The KL algorithm accepts an undirected graph G = (V, E), a label set L, and a family of connected subsets of nodes. Cost c and initial (feasible) solution (x) 0 ,y 0 ), and regarding the current cluster x t-1 The algorithm iterates over all possible pairs (A, B) of adjacent (connected) components (i.e., there exist e = {v, w} ∈ E such that v ∈ A and w ∈ B) (line 3). The algorithm solves a series of 2-cut problems between component pairs (line 4). For the current cluster x... t For each component (line 5), a new component can be introduced into the decomposition by solving the 2-cut edge problem over an empty set (line 6). This process can continue up to the maximum number of iterations, or until there are no further differences in subsequent solutions (line 2). Lines 4 and 6 both rely on the function update, which operates only on a pair of components, and is described below.

[0106]

[0107]

[0108] The function above works by considering only two components. (B can be an empty set) Update the bidirectional clustering to seek a lower target value. The algorithm calculates the cumulative total gain S for all steps 1≤i≤|A∪B|. iIt also maintains a move queue M to keep track of update steps. In each iteration (lines 6-23), it selects a vertex whose movement (possibly, relabeling) to another set will result in the maximum reduction of the objective value (line 7). For efficiency purposes, these values ​​can be pre-computed with respect to the current solution (x, y) using the Kernighan-Lin algorithm (lines 3-5). For any set and any annotation λ U Notes The cost can be constrained with respect to the current (feasible) solution (x, y), where the label of the node u∈U is fixed as l, i.e.,

[0109]

[0110] Where [·] represents Iverson's bracket, in other words:

[0111]

[0112] any The sum of the costs of any node v∈V and label l∈L can be limited to:

[0113]

[0114] in For all v∈A∪B and l∈L, the pre-computed values ​​can be calculated as follows:

[0115]

[0116] In the main loop (lines 6-23), the algorithm selects a vertex v* for potential movement and chooses its new (possibly the same) label (line 7), calculates the actual cumulative gain (line 8) and records this movement (line 9), then updates the D of all other vertices that share the cost with v*. vl The pre-calculated values ​​are (lines 10-23). ​​Different cases are captured in lines 12-23; without loss of generality, assume v* moves from A to B. Note that only a single node is considered in each iteration. Note that suboptimal (i.e., non-decreasing) individual moves in this manner can be allowed in the hope of leaving a local minimum and obtaining a better local minimum later. We search for the step i* that produces the maximum cumulative gain (line 24). A special operation calculates the reduction of the objective of connecting the two components A and B while keeping their node labels y constant (line 25), as follows:

[0117] Δ join =T A +T B -TA∪B (6)

[0118] Among them, for any

[0119]

[0120] This is important because local movement of individual components does not result in connecting A and B, even though it would be beneficial. Choose the optimization operation (line 26): connect the two components A and B (lines 27-28), or keep some movement up to i* (lines 30-36). Then, update the clustering (lines 32-33) and labels (lines 34-36).

[0121]

[0122]

[0123] The GAEC algorithm begins by initially decomposing all vertices v∈V into individual components (line 1). Initially, all nodes are unlabeled (line 2). The algorithm greedily shrinks pairs or three vertices while fixing their node labels. The remaining costs (the shrunken vertices are a portion of the remaining costs) are summed. A minimum priority queue Q is used to keep track of the subset of vertices most likely to shrink. We initialize Q by finding the optimal cost (including node labels) for each pair and each set of three vertices (lines 3-5). The main loop continues until the queue is empty or contains no negative cost elements (line 6). In each iteration, the optimal subset U and its label λ can be selected. U (Line 7). Without loss of generality, we assume that node u of U remains in the graph (lines 8-10), and other vertices in U\{u} shrink along with it (lines 11-24). Lines 15-23 update all costs, and line 16 handles the collapse of cost. The new expected improvement in the objective value is pushed into queue Q. Finally, based on unary cost, the vertices that have not yet been assigned node labels are selected as the cheapest vertices (lines 25-28) and retained in the individual cluster. The unary cost can be defined as... Where u represents a single node.

[0124] Figure 3A The input point cloud 200 representing stars is shown. Specifically, input points 200 represent four "Big Dipper" constellations, four Cassiopeia star clusters, and 52 outlier values. Typically, such input points 200 can be constrained to 2D coordinates in Euclidean coordinates and can be obtained directly as 2D coordinates using this system and method, or by detecting stars in night sky images.

[0125] Figure 3B An example of object recognition of a star cluster is illustrated, which is applied to input point 200 to identify stars based on a geometric template of a defined star cluster. Figure 3B In this embodiment, multiple star clusters 210 are identified as point clusters, where the points in each cluster are interconnected by edges in the corresponding graph. Figure 3B In the diagram, this edge connectivity is shown by thin lines, while the outline of the star clusters, based on the obtained annotations, is shown as thick lines.

[0126] In another implementation, keypoints can be detected in image data acquired by an onboard camera of the vehicle using one or more keypoint detectors. For example, corners or similar points can be detected as keypoints. These keypoints can be converted into input data for geometric object recognition by back-projecting the extracted keypoints into a 3D coordinate system based on depth information from the image data. This depth information can be obtained from, for example, a stereo camera system or an additional depth sensor, or it can be estimated. Geometric object detection can then be applied to these points based on a geometric template of the vehicle to derive inferred keypoints. This vehicle recognition may involve obtaining CAD models of multiple vehicle types. Input points in 3D Euclidean space can be obtained by back-projecting keypoints extracted from a 2D digital image into a 3D coordinate system using a camera matrix and the estimated or measured depth information. Object recognition can then efficiently search the CAD model in the input point cloud using a scale-invariant similarity metric, assuming a similarity transformation has occurred. One possible application area is driver assistance systems that operate when the vehicle is traveling on a highway. Here, the car's orientation changes relatively little, but the car's zoom level varies significantly during operation due to the different distances between the car and the camera.

[0127] The object recognition in the geometric domain described above can also be applied to several other application areas, such as articulated object detection. Here, a graph G = (V, E) with arbitrary structure E can be used to model a composite object consisting of rigid parts that can move around a joint point. This movement can be modeled as a piecewise rigid transformation. One embodiment of an articulated object is a moving robotic arm. The geometry of the robotic arm is known, so the geometric template is readily available. By using markers on the joint point, the input point can be obtained in the form of the marker's location. Even for partially obscured parts, the exact orientation of the robotic arm can be determined through geometric object recognition. This application can be incorporated into safety systems that require the identification of hazardous situations. For marker extraction, IR or UV cameras can be used, depending on given conditions, to increase the robustness of the system. A similar application area is the use of geometric object recognition to inspect parts (e.g., pistons, valves).

[0128] Another application example is traffic sign recognition. Here, planar geometric data or image data defining multiple traffic signs can be obtained. To model traffic signs from this type of data, one or more types of points of interest, such as corners, can be extracted from the data. By extracting the same type of points of interest from sensor data, an input point cloud can be obtained from sensor data (such as image data acquired by a vehicle's onboard camera).

[0129] Another application example is character recognition. Here, planar geometric data or image data can be obtained, which defines a finite set of characters, thereby defining a set of 2D objects. To model characters from this type of data, one or more types of points of interest, such as corners, can be extracted from the data. By extracting the same type of points of interest from sensor data, an input point cloud can be obtained from sensor data (such as image data of a document page acquired by a scanner or digital camera). In the case of a digital camera, if the camera's orientation is not perpendicular to the plane of the document page, the transformation is a stereo projection, which can be interpreted by a scale-invariant similarity metric.

[0130] Another application example is the detection and identification of small objects with known geometries. This might relate to robots that can be configured to find and grasp objects. Such robots can also be configured to count objects. By being able to identify objects, counting them becomes irrelevant.

[0131] Figure 4A computer-implemented method 400 for object recognition is illustrated, which is a high-level representation of some embodiments described in this specification. Method 400 may include accessing template data representing a geometric template in a step entitled "Accessing Template Data" (410), accessing input data for object recognition in a step entitled "Accessing Input Data" (420), providing a cost function in a step entitled "Providing Cost Function" (430), performing greedy additive edge contraction (GAEC) in a step entitled "Greedy Additive Edge Contraction (GAEC)" (440), performing the Kernighan-Lin (KL) algorithm in a step entitled "KERNIGHAN-LIN (KL) Algorithm" (450), and outputting object recognition data representing clusters and labels of nodes in a step entitled "Outputting Clusters and Node Labels" (460).

[0132] The method 400 and any other methods, algorithms, or pseudocode described in this specification may be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. Furthermore... Figure 5 As illustrated, instructions for a computer (e.g., executable code) may be stored on a computer-readable medium 500, for example, in the form of a series of machine-readable physical marks 510 and / or as a series of elements having different electrical (e.g., magnetic or optical) properties or values. The executable code may be stored in a transient or non-transient manner. Embodiments of the computer-readable medium include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 5 The image shows CD 500.

[0133] Examples, implementations, or optional features, whether or not indicated as non-limiting, should not be construed as limiting the scope of the claimed invention.

[0134] It should be noted that the above embodiments are illustrative and not limiting of the invention, and many alternative embodiments will be able to be devised by those skilled in the art without departing from the scope of the appended claims. Any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The use of the verb “comprising” and its variations does not exclude the presence of elements or stages other than those described in the claims. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Phrases such as “at least one” preceding a list or group of elements represent the selection of all or any subset of elements from the list or group. For example, the phrase “at least one of A, B, and C” should be understood to include: only A; only B; only C; both A and B; both A and C; both B and C; or all A, B, and C. The invention can be embodied by hardware comprising several different elements and by a suitably programmed computer. In the device claims enumerating several means, several means of these means may be embodied by the same item of hardware. The mere fact that certain measures are described in different dependent claims does not imply that a combination of these measures cannot be used to gain an advantage.

Claims

1. A computer-implemented method (400) for object recognition, comprising: - Access (410) includes template data including template points, the template points being defined as coordinates in a coordinate system, wherein the template points represent the geometric template of an object, and wherein the template points are labeled to distinguish corresponding parts of the object; - Access sensor data acquired by sensors, wherein the sensors are camera sensors, depth sensors, lidar sensors, or radar sensors; - Generate input data for object recognition, the input data including input points defined as coordinates in the coordinate system, wherein the input points are generated by extracting points of interest from the sensor data; - Construct an undirected weighted graph having nodes representing the input point and edges between the nodes; - Object recognition is performed by finding one or more occurrences or partial occurrences of the object in the input data based on the following steps: - Assign edge labels to the edges to define the cut edges in the graph, thereby decomposing them into one or more clusters representing objects, and - Assign node labels from a set of labels to the node to identify the corresponding node as representing the corresponding part of the corresponding object. The edge and node annotations include: - Provide a (430) cost function that expresses the scale-invariant similarity between the corresponding triangle in the node and the corresponding triangle in the template point, wherein the scale-invariant similarity contributes to the cost function if the label of the node matches the label of the template point and the node is part of the same cluster; - Apply a first local search algorithm (440) to an initial graph having each node unlabeled and each edge labeled to define a cut edge, wherein the first local search algorithm is configured to iteratively connect nodes by edge labeling to form clusters in the graph and to search for an initial local minimum in the cost function by iteratively labeling nodes, thereby obtaining initial edge labels and initial node labels; - Apply a second local search algorithm (450) to the initial edge labeling and initial node labeling, wherein the second local search algorithm is configured to find an improvement of the initial local minimum of the cost function by iteratively moving nodes between clusters through edge relabeling and node relabeling; - Generate (460) object identification data as output, the object identification data representing the assigned edge labels and the assigned node labels.

2. The computer-implemented method (400) according to claim 1, wherein the cost function expresses the similarity between corresponding triangles as a function defining the angles of the triangles.

3. The computer-implemented method (400) according to claim 2 further includes determining the similarity as a function defining the angle of the corresponding triangle by projecting the angle of the triangle as 3D coordinates onto a 2D simplex and by calculating the distance between the coordinates of the triangle projected onto the 2D simplex.

4. The computer-implemented method (400) according to any one of claims 1 to 3, wherein the first local search algorithm is a greedy additive edge contraction algorithm.

5. The computer-implemented method (400) according to any one of claims 1 to 3, wherein the second local search algorithm is the Kernighan-Lin algorithm.

6. The computer-implemented method (400) according to any one of claims 1 to 3, wherein the geometric template data includes template points representing a composite geometric template of multiple objects, wherein the template points are labeled to distinguish corresponding portions of corresponding objects among the multiple objects.

7. The computer-implemented method (400) according to any one of claims 1 to 3, wherein the sensor data is image data acquired by a camera, and the method includes generating the input point by extracting the point of interest from the image data as a key point.

8. The computer-implemented method (400) of claim 7, wherein the image data is 2D image data, and the method includes generating the input point by extracting the key points from the 2D image data and back-projecting the extracted key points into a 3D coordinate system based on depth information of the 2D image data.

9. The computer-implemented method (400) according to any one of claims 1 to 3, wherein the template points are labeled by continuously indexing the template points within the coordinate system.

10. A computer-readable medium (500) comprising temporary or non-temporary data (510) representing instructions arranged to cause a processor system to perform a computer-implemented method according to any one of claims 1 to 9.

11. A system (100) for object recognition, comprising: - Sensor interface (120), configured for: Access sensor data (122) acquired by sensor (20), wherein the sensor is a camera sensor, depth sensor, lidar sensor or radar sensor; - An input interface, said input interface including a data storage interface (180), configured for: Access template data (192), the template data including template points, the template points being defined as coordinates in a coordinate system, wherein the template points represent the geometric template of an object, wherein the template points are labeled to distinguish corresponding parts of the object; - The processor subsystem (160) is configured for: Input data for object recognition is generated, the input data including input points defined as coordinates in the coordinate system, wherein the input points are generated by extracting points of interest from the sensor data; Construct an undirected weighted graph having nodes representing the input point and edges between the nodes; Object recognition is performed by finding one or more occurrences or partial occurrences of the object in the input data based on the following steps: - Assign edge labels to the edges to define the cut edges in the graph, thereby obtaining one or more clusters representing the objects, and - Assign node labels from a set of labels to the nodes to identify the corresponding nodes as representing the corresponding parts of the corresponding objects. The edge and node annotations include: - Provide a cost function that expresses the scale-invariant similarity between the corresponding triangle in the node and the corresponding triangle in the template point, wherein the scale-invariant similarity contributes to the cost function if the label of the node matches the label of the template point and the node is part of the same cluster; - Apply a first local search algorithm to an initial graph having each node unlabeled and each edge labeled to define a cut edge, wherein the first local search algorithm is configured to iteratively connect nodes by edge labeling to form clusters in the graph and to search for an initial local minimum in the cost function by iteratively labeling nodes, thereby obtaining initial edge labels and initial node labels; - An improvement to the initial local search algorithm is applied to the initial edge labeling and initial node labeling, wherein the second local search algorithm is configured to iteratively move nodes between clusters by relabeling edges and nodes to find an initial local minimum of the cost function; - An output interface for outputting object identification data (196), the output interface including the data storage interface (180), the object identification data representing assigned edge labels and assigned node labels.

12. The system (100) of claim 11, wherein the sensor data is image data acquired by a camera, and the processor subsystem is configured to generate the input points by extracting the points of interest from the image data as key points.

13. The system (100) of claim 11 or 12 further includes an actuator interface (140) for providing control data (142) to the actuator (40), wherein the processor subsystem (160) is configured to generate the control data based on the object identification data to control the actuator.

14. A vehicle (80) or robot comprising the system (100) according to claim 11 or 12.

Images