Systems and methods for training a model to predict dense correspondences in images using geodesic distances

Abstract

Systems and methods for training a model to predict dense correspondences across images, such as human images. The model can be trained using synthetic training data created from one or more 3D computer models of a subject. Additionally, one or more geodesic distances derived from a surface of one or more of the 3D models can be used to generate one or more loss values that, in turn, are used to modify parameters of the model during training.

Description

Background Technology

[0001] Various tasks in computer vision rely on systems configured to recognize correspondences across images. This can be particularly challenging in certain contexts, such as images of people, which can vary widely based on the object's body orientation, clothing, camera angle, etc., and can contain confusing similarities, such as between an object's right and left hands. This can also be a challenge for other types of objects, such as animals, robots, or mechanical devices. In some approaches, models can be trained to recognize similar features using images manually encoded by human annotators (e.g., for images of people, each image can be encoded to recognize eyes, ears, nose, shoulders, elbows, hands, etc.).

[0002] However, because it is impractical for human annotators to identify every pixel in every reference image, and because humans may not be able to identify features precisely enough, models interpreting training examples with human annotations often have to be configured to draw additional conclusions by associating pixels around each identified feature as more likely to be associated with that feature. This can lead to learning incorrect associations. For example, using training data with human annotations, if a human subject's head is tilted so that their cheek rests on their shoulder, nearest neighbor search might incorrectly identify pixels of the subject's cheek as corresponding to the subject's shoulder, or vice versa. This can cause the model to misinterpret human forms in other images during inference. An additional drawback of using training examples with human annotations is the high cost of creating such training data. Summary of the Invention

[0003] This technology relates to systems and methods for training models to predict dense correspondences across images such as human images. More specifically, this technology provides training models using synthetic training data created from one or more three-dimensional computer models (“3D models”), such as 3D models of human objects. Additionally, this technology provides generating one or more loss values ​​for modifying model parameters during training using one or more geodesic distances derived from the surfaces of one or more of the 3D models.

[0004] This technique allows for the automatic creation of virtually unlimited amounts and types of training data, and additionally enables the integration of geodesic surface information from 3D models into the training data, allowing the predictive model to learn to accurately distinguish features that might otherwise appear relevant in two-dimensional images (“2D images”). Therefore, predictive models trained according to this technique can achieve significantly lower error rates when predicting correspondences between pairs of images—including between points (e.g., pixels) visible in two images—and when identifying when a point visible in one image is occluded and therefore invisible in another. Furthermore, predictive models trained according to this technique can make fewer errors in corresponding points on blurred features in two images (e.g., between the left and right hands of a human subject). These benefits can be seen in both predicting correspondences between images of a single object and predicting correspondences between images of two different objects. Additionally, predictive models trained according to this technique can be integrated into other end-to-end network architectures that rely on predicting correspondences between images (e.g., models trained to perform motion estimation or “optical flow,” human pose estimation, etc.), resulting in ensemble models that demonstrate the benefits of this technique and achieve better performance.

[0005] In one aspect, this disclosure describes a method for training a neural network to predict corresponding features in an image, the method comprising: generating a first feature map based on a first image of an object using a neural network by one or more processors of a processing system, and generating a second feature map based on a second image of the object, the first and second images being different and generated using a three-dimensional model of the object; determining a first feature distance between a first point represented in the first feature map and a second point represented in the second feature map by one or more processors, the first and second points corresponding to the same feature on the three-dimensional model of the object; determining a second feature distance between a third point represented in the first feature map and a fourth point represented in the first feature map by one or more processors; and determining a second feature distance between a third point represented in a first feature map and a fourth point represented in the second feature map by one or more processors. A first geodesic distance between four points, a first surface map corresponding to a first image and generated using a 3D model of the object; a third feature distance determined by one or more processors between a third point represented in the first feature map and a fifth point represented in the first feature map; a second geodesic distance determined by one or more processors between the third point and the fifth point represented in the first surface map; a first loss value determined by one or more processors from a set of loss values, the first loss value being based on the first feature distance; a second loss value determined by one or more processors from the set of loss values, the second loss value being based on the second feature distance, the third feature distance, the first geodesic distance, and the second geodesic distance; and one or more processors modifying one or more parameters of the neural network based at least in part on the set of loss values. In some aspects, the first loss value is further based on a set of additional feature distances, each given feature distance in the set of additional feature distances being between a selected point represented in the first feature map and a corresponding point represented in the second feature map, the selected point and the corresponding point corresponding to the same feature on the 3D model of the object. In some aspects, the first point and each selected point together represent all pixels in the first image. In some respects, the second loss value is further based on at least one additional feature distance pair and at least one additional geodesic distance pair, each of the at least one additional feature distance pair comprising two feature distances between a set of three selected points represented in the first feature map, and each of the at least one additional geodesic distance pair comprising two geodesic distances between a set of three selected points represented in the first surface map.In some aspects, the method further includes: determining, by one or more processors, a set of fourth feature distances between a sixth point represented in a first feature map and all other points of a first image represented in the first feature map; determining, by one or more processors, a set of third geodesic distances between the sixth point represented in a first surface map and all other points of the first image represented in the first surface map; and determining, by one or more processors, a third loss value in a set of loss values, the third loss value being based on the set of fourth feature distances and the set of third geodesic distances. In some aspects, the third loss value is further based on at least one additional set of feature distances and at least one additional set of geodesic distances, each given additional set of feature distances in the at least one additional set of feature distances being between a selected point represented in the first feature map and all other points of the first image represented in the first feature map, and each given additional set of geodesic distances in the at least one additional set of geodesic distances being between a selected point represented in the first surface map and all other points of the first image represented in the first surface map. In some aspects, the method further includes: determining, by one or more processors, a set of fourth feature distances between a sixth point represented in a first feature map and all points of a second image represented in a second feature map; determining, by one or more processors, a set of third geodesic distances between a first point represented in a second surface map and all points of a second image represented in the second surface map, the second surface map corresponding to the second image and generated using a three-dimensional model of the object, and the first point in the second surface map and the sixth point in the first feature map corresponding to the same features on the three-dimensional model of the object; and determining, by one or more processors, a third loss value in a set of loss values, the third loss value being based on the set of fourth feature distances and the set of third geodesic distances. In some respects, the third loss value is further based on at least one additional set of feature distances and at least one additional set of geodesic distances, each given additional set of feature distances in the at least one additional set of feature distances being between the selected point represented in the first feature map and all points of the second image represented in the second feature map, and each given additional set of geodesic distances in the at least one additional set of geodesic distances being between the corresponding point represented in the second surface map and all points of the second image represented in the second surface map, the corresponding point in the second surface map and the selected point in the first feature map corresponding to the same feature on the three-dimensional model of the object.In some aspects, the method further includes: determining, by one or more processors, a set of fifth feature distances between a seventh point represented in a first feature map and all points in a second image represented in a second feature map; determining, by one or more processors, a set of fourth geodesic distances between a first point represented in a second surface map and all points in a second image represented in the second surface map, the second surface map corresponding to the second image and generated using a three-dimensional model of the object, and the first point in the second surface map and the seventh point in the first feature map corresponding to the same features on the three-dimensional model of the object; and determining, by one or more processors, a fourth loss value in a set of loss values, the fourth loss value being based on the set of fifth feature distances and the set of fourth geodesic distances. In some aspects, the first point represented in the second surface map corresponds to a feature not represented in the second feature map on the three-dimensional model of the object. In some aspects, the method further includes one or more processors generating at least one of a first image, a second image, or a first surface map. In some aspects, the method further includes one or more processors generating at least one of a first image, a second image, a first surface map, or a second surface map. In some aspects, the object is a human or a representation of a human. In some aspects, the object is in a pose in the first image that differs from its pose in the second image. In some respects, the first image is generated from a different perspective of the object's three-dimensional model compared to the second image.

[0006] In another aspect, this disclosure describes a processing system including a memory storing a neural network; and one or more processors coupled to the memory and configured to use the neural network to predict correspondences in an image, wherein the neural network has been trained according to a training method to predict correspondences in the image, the training method comprising: generating a first feature map based on a first image of an object, and generating a second feature map based on a second image of the object, the first and second images being different and generated using a three-dimensional model of the object; determining a first feature distance between a first point represented in the first feature map and a second point represented in the second feature map, the first and second points corresponding to the same feature on the three-dimensional model of the object; and determining the first feature map. The method involves: determining a second feature distance between a third and a fourth point represented in a first surface map; determining a first geodesic distance between a third and a fourth point represented in a first surface map, the first surface map corresponding to a first image and generated using a 3D model of the object; determining a third feature distance between a third point and a fifth point represented in the first feature map; determining a second geodesic distance between a third and a fifth point represented in the first surface map; determining a first loss value in a set of loss values, the first loss value being based on the first feature distance; determining a second loss value in a set of loss values, the second loss value being based on the second feature distance, the third feature distance, the first geodesic distance, and the second geodesic distance; and modifying one or more parameters of the neural network, at least in part, based on the set of loss values. In some aspects, the neural network has been trained according to a training method to predict correspondences in an image, the training method further comprising: determining a set of fourth feature distances between a sixth point represented in a first feature map and all other points of a first image represented in the first feature map; determining a set of third geodesic distances between a sixth point represented in a first surface map and all other points of the first image represented in the first surface map; and determining a third loss value in a set of loss values, the third loss value being based on the set of fourth feature distances and the set of third geodesic distances. In some aspects, the neural network has been trained according to a training method to predict correspondences in an image, the training method further comprising: determining a set of fourth feature distances between a sixth point represented in a first feature map and all points of a second image represented in a second feature map; determining a set of third geodesic distances between a first point represented in a second surface map and all points of the second image represented in the second surface map, the second surface map corresponding to the second image and generated using a three-dimensional model of the object, and the first point in the second surface map and the sixth point in the first feature map corresponding to the same features on the three-dimensional model of the object; and determining a third loss value in a set of loss values, the third loss value being based on the set of fourth feature distances and the set of third geodesic distances.In some aspects, the neural network has been trained according to a training method to predict correspondences in an image, the training method further comprising: determining a set of fifth feature distances between a seventh point represented in a first feature map and all points of a second image represented in a second feature map; determining a set of fourth geodesic distances between a first point represented in a second surface map and all points of a second image represented in the second surface map, the second surface map corresponding to the second image and generated using a three-dimensional model of the object, and the first point in the second surface map and the seventh point in the first feature map corresponding to the same features on the three-dimensional model of the object; and determining a fourth loss value in a set of loss values, the fourth loss value being based on the set of fifth feature distances and the set of fourth geodesic distances. Attached Figure Description

[0007] Figure 1 It is a functional diagram of an example system based on various aspects of this disclosure.

[0008] Figure 2 It is a functional diagram of an example system based on various aspects of this disclosure.

[0009] Figure 3 It is a graph that illustrates how points in a given 2D image can be represented in the corresponding feature map and 3D surface map;

[0010] Figure 4A and Figure 4B This is a flowchart illustrating an exemplary method for training a neural network to identify corresponding features in an image, according to various aspects of the present technology.

[0011] Figure 5 This illustrates various aspects of the present technology for determining additional loss values ​​for use with... Figure 4A , 4B A flowchart of an exemplary method for combining loss values ​​of methods 400a and 400b.

[0012] Figure 6 This illustrates various aspects of the present technology for determining additional loss values ​​for use with... Figure 4A , 4B A flowchart of an exemplary method for combining loss values ​​of methods 400a and 400b.

[0013] Figure 7 This illustrates various aspects of the present technology for determining additional loss values ​​for use with... Figure 4A , 4B A flowchart of an exemplary method for combining loss values ​​of methods 400a, 400b and 500. Detailed Implementation

[0014] This technology will now be described with respect to the following exemplary systems and methods.

[0015] Figure 1 A high-level system diagram 100 is shown for an exemplary processing system for performing the methods described herein. The processing system 102 may include one or more processors 104 and a memory 106. The memory 106 stores information accessible by the one or more processors 104, including instructions 108 and data 110 that can be executed or otherwise used by the processors 104. The memory 106 may be any non-transitory type capable of storing information accessible by the processors 104. For example, the memory 106 may include non-transitory media such as hard disk drives, memory cards, optical discs, solid-state drives, magnetic tape storage, etc. The instructions 108 and data 110 may include a prediction model, training data (e.g., 2D images and corresponding 3D model data), and a 3D model for creating the training data, as described herein.

[0016] Processing system 102 can be implemented on any type of computing device, such as any type of general-purpose computing device, server, or a combination thereof, and may further include other components typically found in general-purpose computing devices or servers. Therefore, processing system 102 can reside on a single computing device such as a server, personal computer, or mobile device, and the model described herein can thus be local to that single computing device. Similarly, processing system 102 can reside on a cloud computing system or other distributed system, such that the predictive model, training data, and / or the 3D model used to create the training data described herein are distributed across two or more different physical computing devices.

[0017] in this regard, Figure 2 An additional high-level system diagram 200 is shown, in which an exemplary processing system 202 for performing the methods described herein is shown as including a first computing device 202a, a second computing device 202b, and a database 202c, each of which is interconnected via one or more networks 216. The first computing device 202a, the second computing device 202b, and the database 202c may each be a single computing device, or they may be distributed across two or more different physical computing devices. Figure 1 Similar to the examples, the first computing device 202a and the second computing device 202b each include one or more processors (204a, 204b) and memories (206a, 206b) that store instructions (208a, 208b) and data (210a, 210b). Although not shown, the database 202c may also include one or more processors and memories that store instructions and data.

[0018] like Figure 2As illustrated in the example, the data 210a of the first computing device 202a stores a prediction model 212 and training data 214a, which includes 2D images and corresponding 3D model data used to train the prediction model 212. The data 210b of the second computing device 202b may store one or more 3D models 218, and a set of training data 214b created using one or more 3D models 218, which includes 2D images and corresponding 3D model data. Finally, the database 202c may also store a set of training data 214c, which also includes 2D images and corresponding 3D model data created by the second computing device 202b using one or more 3D models 218. In this respect, the sets of training data 214a, 214b, and 214c may be identical to each other, or one or more may be a subset of the others. For example, the second computing device 202b can be configured to create training data 214b from one or more 3D models 218 and transmit the training data to a database 202c for storage in a master set 214c. Furthermore, the first computing device 202a can be configured to obtain some or all of the training data 214c from the database 202c and store it locally in the memory 206a of the first computing device 202a as training data 214a for training the prediction model 212.

[0019] In all cases, the computing device described herein may further include any other components typically used in conjunction with the computing device, such as a user interface subsystem. The user interface subsystem may include one or more user inputs (e.g., a mouse, keyboard, touchscreen, and / or microphone) and one or more electronic displays (e.g., a monitor with a screen or any other electronic device operable to display information). Output devices other than electronic displays, such as speakers, lights, and vibrational, pulsating, or haptic elements, may also be included in the computing device described herein.

[0020] The one or more processors included in each computing device can be any conventional processor, such as a commercially available central processing unit (“CPU”), graphics processing unit (“GPU”), tensor processing unit (“TPU”), etc. Alternatively, the one or more processors can be special-purpose devices, such as ASICs or other hardware-based processors. Each processor may have multiple cores capable of operating in parallel. The processor, memory, and other components of a single computing device may be stored within a single physical enclosure or may be distributed across two or more enclosures. Similarly, the memory of a computing device may include hard disk drives or other storage media located in an enclosure different from the processor enclosure, such as in an external database or networked storage device. Therefore, references to processors or computing devices will be understood to include a collection of processors or computing devices or memory that may or may not operate in parallel, as well as one or more servers in a load-balanced server cluster or cloud-based system.

[0021] The computing device described herein can store instructions that can be executed directly (such as machine code) or indirectly (such as scripts) by a processor. The computing device can also store data that can be retrieved, stored, or modified by one or more processors according to the instructions. Instructions can be stored as computing device code on a computing device-readable medium. For this purpose, the terms "instruction" and "program" are used interchangeably herein. Instructions can also be stored in object code format for direct processor processing, or in any other computing device language, including collections of scripts or standalone source code modules that are interpreted on demand or compiled in advance. For example, the programming language could be C#, C++, JAVA, or another computer programming language. Similarly, any component of the instructions or program can be implemented using a computer scripting language, such as JavaScript, PHP, ASP, or any other computer scripting language. Furthermore, any of these components can be implemented using a combination of computer programming languages ​​and computer scripting languages.

[0022] like Figure 2As illustrated in the example, the data 210a of the first computing device 202a stores a prediction model 212 and training data 214a, which includes 2D images and corresponding 3D model data used to train the prediction model 212. The data 210b of the second computing device 202b may store a set of one or more 3D models 218 and training data 214b, which includes 2D images and corresponding 3D model data created using one or more 3D models 218. Finally, the database 202c may also store a set of training data 214c, which also includes 2D images and corresponding 3D model data created by the second computing device 202b. In this respect, the sets of training data 214a, 214b, and 214c may be identical to each other, or one or more may be a subset of the others. For example, the second computing device 202b can be configured to create training data 214b from one or more 3D models 218 and transmit the training data to a database 202c for storage in a master set 214c. Furthermore, the first computing device 202a can be configured to obtain some or all of the training data 214c from the database 202c and store it locally in the memory 206a of the first computing device 202a as training data 214a for training the prediction model 212.

[0023] Figure 3 The illustration shows how points in a given 2D image can be represented in the corresponding feature map and 3D surface map. In this regard, Figure 3 A 2D image 302 generated from a 3D model of a human object (not shown) is shown. The 3D model can also be used to generate a corresponding surface map 306 of an object in the same pose as the one used to generate the 2D image 302. Figure 3 The image depicts two segments 306a and 306b of a complete 3D surface map 306. Image 302 can be processed, for example, using a learned embedding function to generate feature map 304. Although feature map 304... Figure 3 While graphically represented, the feature map F can actually take any suitable form and dimension, such as a database, tensor, vector, etc. For example, for an image with a height H in pixels and a width W in pixels, the feature map F can be expressed according to the following Equation 1:

[0024]

[0025] In this case, each pixel of the image will be represented as a vector of dimension C in the feature map F, where dimension C can be any suitable integer (e.g., 1, 16, etc.).

[0026] exist Figure 3In the example, three points A, B, and C are identified in each of the 2D image 302, feature map 304, and 3D surface image segments 306a and / or 306b. Again, although these points are graphically represented in feature map 304, they can actually be represented alternatively as individual values ​​in a database or as distinct elements in a tensor or vector. It can be seen that feature map 304 can be used to determine the first feature distance d between point A and point B. A,B And the second characteristic distance d between points B and C B,C These feature distances can be determined using any suitable equation. For example, if each point in the 2D image 302 is represented as a separate vector in the feature map 304, then the feature distance between the vectors representing points A and B can be determined by calculating the dot product of the two vectors, by calculating the cross product of the two vectors, by adding the two vectors, by subtracting the two vectors, etc. In this regard, for the feature map F described above with respect to Equation 1, the feature distance d between point p in image I1 and point q in image I2 can be calculated according to the following Equation 2:

[0027] d(p,q)=1-F1(p)·F2(q) (2)

[0028] In some respects, the feature vectors F1(p) and F2(q) representing point p in image I1 and point q in image I2, respectively, can first be normalized so that they are unit vectors.

[0029] Similarly, 3D surface map 306 can be used to determine the first geodesic distance g between point A and point B. A,B And the second geodesic distance g between points B and C B,C Geodesic distance represents the distance between those points measured along a 3D surface, and can be determined based on any suitable equation or method used to calculate (or estimate) the shortest distance between those points on a 3D surface.

[0030] Figure 4A and 4BExemplary methods 400a and 400b are depicted, illustrating how a neural network can be trained by a processing system (e.g., processing system 102 or 202) to recognize correspondences in an image. In this regard, in step 402, a first image and a first surface map are generated from a 3D model of the object. Similarly, in step 404, a second image different from the first image is generated from the 3D model. The first image may be generated in a first pose and / or oriented from a first camera, and the corresponding first surface map will at least represent the surface of the 3D model in that first pose. The second image can be generated in any way that creates an image different from the first image. For example, it can be generated by placing the 3D model in a second pose, by imaging the 3D model from different perspectives (e.g., generating images from different virtual camera positions), by subjecting the 3D model to lighting conditions different from those used to generate the first image, etc. The first image, the first surface map, and the second image may be generated by a processing system (e.g., processing system 102 or 202) used to train the neural network, or they may be generated by a separate system and provided to the processing system used to train the neural network.

[0031] In step 406, the processing system generates a first feature map from the first image and a second feature map from the second image. (Refer to the above...) Figure 3 As explained, the processing system can generate feature maps in any suitable manner. For example, the processing system can use a learned embedding function to generate a database, vector, or tensor that includes entries, elements, or other values ​​representing each point (e.g., pixel) in a given image.

[0032] In step 408, the processing system determines a first feature distance between a first point represented in the first feature map and a second point represented in the second feature map, the first point and the second point corresponding to the same feature on the 3D model. For example, if the first and second images are human objects, the first point in the first image and the second point in the second image may both correspond to the fingertip of the object's right index finger. Alternatively, if each point in the first and second images is represented as a vector in the first and second feature maps respectively, the first feature distance can be determined using the vector representing the first point in the first feature map and the vector representing the second point in the second feature map as described above with respect to Equation 2. Similarly, as mentioned above, the first feature distance can be determined using any other suitable equation, such as by calculating the straight dot product of the two vectors, by calculating the cross product of the two vectors, by adding the two vectors, by subtracting the two vectors, etc.

[0033] In step 410, the processing system determines a first loss value from the set of loss values ​​based on the first feature distance. This first loss value can be determined according to any suitable equation. Similarly, the first loss value can be determined for any number of point pairs in the first and second images. For example, the first loss value can be the consistency loss value L calculated according to Equation 3 below. c The feature distance d is calculated between each point p (a total of n points) in the first image and the corresponding point corr(p) in the second image.

[0034]

[0035] In the example of Equation 3, the loss value L is calculated based on the first feature distance and all other feature distances between corresponding point pairs in the first and second images. c In this respect, the relationship between each point p in the second image and its corresponding point corr(p) can be determined by the processing system by associating each point p with its corresponding point in the 3D model, and then associating that point in the 3D model with its corresponding corr(p) in the second image. Alternatively, in some aspects, the training data can further include a list or other data structure that identifies all corresponding point pairs, so that the processing system does not need to make these determinations.

[0036] In step 412 ( Figure 4B In step 414, the processing system determines a second feature distance between the third point represented in the first feature map and the fourth point represented in the first feature map. Similarly, in step 414, the processing system determines a third feature distance between the third point represented in the first feature map and the fifth point represented in the first feature map. For example, the third point can be any randomly selected reference point p in the first image. r Furthermore, the fourth and fifth points can be any other randomly selected point p in the first image. a and p b Similarly, if each point in the first image is represented as a vector in the first feature map, then the second and third feature distances can be determined as described above with respect to Equation 2. Likewise, as mentioned above, the second and third feature distances can be determined using any other suitable equation, such as by calculating the dot product of each vector pair, by calculating the cross product of each vector pair, by adding each vector pair, by subtracting each vector pair, etc. In some respects, the equations used to calculate the second and third feature distances can differ from the equations used to calculate the first feature distance.

[0037] In step 416, the processing system determines a first geodesic distance between the third and fourth points represented in the first surface map. Similarly, in step 418, the processing system determines a second geodesic distance between the third and fifth points represented in the first surface map. The first surface map corresponds to a first image, which has been generated using a 3D model of an object in the same pose used to generate the first image. Therefore, using the exemplary points indicated in the descriptions of steps 412 and 414, the third point represented in the first surface map will correspond to point p represented in the first image. r Correspondingly, the fourth and fifth points in the first surface image will correspond to point p in the first image, respectively. a and p b Correspondingly, in this case, the first geodesic distance g(p) r p a ) and the second geodesic distance g(p) r p b ) represent the distances between those points measured along the 3D surface of the first surface map.

[0038] In step 420, the processing system determines a second loss value from the set of loss values ​​based on the second feature distance, the third feature distance, the first geodesic distance, and the second geodesic distance. For example, the second loss value can be configured to encourage two points p in the feature space. a and p b Relative to reference point p r The order of the measurements is the same as that measured by the corresponding geodesic distances, such that physically separated pairs of points on the 3D surface tend to have larger feature distances. This second loss value can be determined according to any suitable equation. For example, the second loss value can be based on the difference between the second and third feature distances, and can be further based on the sign (also known as the signum or sgn function) of the difference between the first and second geodesic distances. In this regard, the second loss value can be the sparse geodesic loss value L calculated according to equations 4 and 5 below. s .

[0039] L s =log(1+exp(s·(d(p)) r p a )-d(p r p d (4)

[0040] s = sgn(g(p) r p a )-g(p r ,pb)) (5)

[0041] Similarly, the second loss value can be based on more than one triple {p} in the first image.r ,p a ,p b Therefore, the second loss value can be based on the following values, vectors, or tensors: the loss value L calculated using the second and third feature distances, as well as the first and second geodesic distances according to Equations 4 and 5. s And the additional loss value L calculated for other randomly selected triples. s For example, the second loss value could be a loss value L calculated for a predetermined number of randomly selected triples—e.g., 1, 16, 128, etc. s The average value. Similarly, the second loss value can be calculated by concatenating the loss value L for a predetermined number of randomly selected triples. s To determine.

[0042] In step 422, the processing system modifies one or more parameters of the neural network, at least in part, based on the set of loss values. In this respect, the neural network can be any model trained to predict dense correspondences across images (e.g., prediction model 212). The set of loss values ​​can be used in any suitable manner to influence the modification of one or more parameters of the neural network. In this respect, the processing system can sum the first and second loss values ​​to obtain a total loss value and modify the parameters of the prediction model based on this total loss value. Similarly, the first and second loss values ​​(or the set thereof) can be multiplied by one or more pre-selected weighting factors (e.g., a specific w) before being combined to form the total loss value. c and w s (Value). In some aspects, the processing system can be configured to use each total loss value directly after each training example, apply it during backpropagation to tune the model's parameters, and then compute a new total loss value during the next training example. In some aspects, the processing system can be configured to batch multiple training examples. In this case, the processing system can be configured to combine (e.g., sum or average) the loss values ​​computed during each training example in the batch, apply the combined total loss value during the backpropagation phase after the batch ends, and then compute a new combined total loss value during the next batch of training examples.

[0043] Figure 5 An exemplary method 500 is described, which determines an additional loss value to be combined with the loss values ​​of methods 400a and 400b of FIG4. In this regard, in step 502, the processing system performs steps 402-420 of methods 400a and 400b as described above.

[0044] In step 504, the processing system determines the set of fourth feature distances between the sixth point represented in the first feature map and all other points in the first image represented in the first feature map. The sixth point can be any randomly selected reference point p in the first image. r Tag p r This is for convenience only, and does not imply that the randomly selected reference point must be the same as the reference point used in step 412 of method 400b. Similarly, if each point of the first image is represented as a vector in the first feature map, the set of fourth feature distances can be determined as described above with respect to Equation 2. In this respect, the set of fourth feature distances will include those using the representation point p r and p t The vector computation for each point p in the first image t Individual feature distance d(p) r ,p t Similarly, as mentioned above, the set of fourth feature distances can be determined using any other suitable equation, such as by calculating the dot product of each vector pair, by calculating the cross product of each vector pair, by adding each vector pair, by subtracting each vector pair, etc. In some respects, the equation used to calculate the set of fourth feature distances may differ from the equation used to calculate the first, second, and / or third feature distances.

[0045] In step 506, the processing system determines a set of third geodesic distances between the sixth point represented in the first surface map and all other points in the first image represented in the first surface map. Here, again, the sixth point represented in the first surface map will be compared with point p in the first image. r Correspondingly, the third set of geodesic distances will therefore include each point p in the first image. t Individual geodesic distance g(p) r ,p t ), which represents p with respect to the first image. r and p t The distance between corresponding points along the 3D surface of the first surface map.

[0046] In step 508, the processing system determines a third loss value from the set of loss values, which is based on the set of fourth feature distances and the set of third geodesic distances. For example, the third loss value can be used to separate feature distances between non-matching points, depending on the geodesic distances between those pixels. Again, the third loss value can be determined according to any suitable equation. For example, the third loss value can be based on the difference between each geodesic distance in the set of third geodesic distances and each corresponding feature distance in the set of fourth feature distances. In some aspects of this technique, the set of third geodesic distances and the set of fourth feature distances can be measured between the selected reference point (or pixel) and every other point (or pixel) in the first image. In this regard, the third loss value can be for the selected point p in the first image according to Equation 6 below. r and each point p t The dense geodesic loss value L calculated from a total of n points. d .

[0047]

[0048] Similarly, the third loss value can be based on more than one selected point p in the first image. r Therefore, the third loss value can be based on the following values, vectors, or tensors: the loss value L calculated according to Equation 6 using the set of fourth feature distances and the set of third geodesic distances. d And for other randomly selected points p r Calculated additional loss value L d For example, the third loss value could be a randomly selected point p for a predetermined number of times—e.g., 1, 5, 16, 128, etc. r Calculated loss value L d The average value. Similarly, this can be achieved by connecting a predetermined number of randomly selected points p. r Calculated loss value L d To determine the third loss value.

[0049] In step 510, the processing system uses the resulting set of loss values ​​to perform step 422 of method 400b, as described above. In this case, the set of loss values ​​includes the first and second loss values ​​determined in steps 410 and 420, and the third loss value determined in step 508. Again, this set of loss values ​​can be used in any suitable manner to influence the modification of one or more parameters of the neural network. In this regard, the processing system can sum the first, second, and third loss values ​​to obtain a total loss value, and modify the parameters of the prediction model based on this total loss value. Similarly, the first, second, and third loss values ​​can be multiplied by one or more pre-selected weighting factors (e.g., a specific w) before being combined to form the total loss value. c w s and w d value).

[0050] Figure 6 An exemplary method 600 is described, which determines an additional loss value to be combined with the loss values ​​of methods 400a and 400b of FIG4. In this regard, in step 602, the processing system performs steps 402-420 of methods 400a and 400b as described above.

[0051] In step 604, the processing system determines a set of fourth feature distances between the sixth point represented in the first feature map and all points in the second image represented in the second feature map. The terms "sixth point" and "set of fourth feature distances" described in method 600 are related to... Figure 5 The scenarios described in Method 500 are different. In Method 600, the sixth point can be any randomly selected reference point p in the first image. r In some respects, the processing system can be configured to select a reference point p that is visible in the first image but not in the second image. r For example, when the first and second images depict human objects walking, the reference point p... r This could correspond to a point on the object's left leg, visible in the first image but covered by the object's right leg in the second image, and therefore invisible thereas. The processing system is configured to select a reference point p that is invisible in the second image. r This allows neural networks to learn how points in a second image correspond to points visible only in the first image (e.g., due to differences in how objects are posed in the two images, camera position, lighting, etc.). Here, the label p is also used. r This is for convenience and does not imply that the selected reference point must be the same as the reference point used in step 412 of method 400b.

[0052] If each point in the first image is represented as a vector in the first feature map, and each point in the second image is represented as a vector in the second feature map, then the set of fourth feature distances can be determined as described above with respect to Equation 2. In this regard, the set of fourth feature distances will include vectors representing points p. r and p t The vector calculation for each point p in the second image t Individual feature distance d(p) r ,p t Similarly, as mentioned above, the set of fourth feature distances can be determined using any other suitable equation, such as by calculating the dot product of each vector pair, by calculating the cross product of each vector pair, by adding each vector pair, by subtracting each vector pair, etc. In some respects, the equation used to calculate the set of fourth feature distances may differ from the equation used to calculate the first, second, and / or third feature distances.

[0053] In step 606, the processing system determines a set of third geodesic distances between the first point represented in the second surface map and all points in the second image represented in the second surface map. The second surface map corresponds to the second image and has been generated using a 3D model of an object in the same pose used to generate the second image. Furthermore, the first point in the second surface map and the sixth point in the first feature map correspond to the same features on the 3D model of the object. For example, as described above, the first point in the second surface map may correspond to a point on the left leg of the object visible in the first image and is therefore represented in the first feature map, but is not visible in the second image and is therefore not represented in the second feature map. Therefore, using the exemplary point indicated in the description of step 604, the first point represented in the second surface map will correspond to point p represented in the first image. r Correspondingly, in this case, the set of third geodesic distances will include each point p for the second image. t The individual geodesic distance g(corr(p) r ),p t ), which represents the distance along the 3D surface of the second surface map between the following points: and point p of the first image. r (Here it is called corr(p)) r The corresponding points and the points p in the second image t Corresponding points.

[0054] In step 608, the processing system determines a third loss value from the set of loss values, the third loss value being based on the set of the fourth feature distances and the set of the third geodesic distances. Similarly, the "third loss value" described with respect to method 600 is also related to... Figure 5The scenario described in Method 500 is different. In this respect, the third loss value of Method 600 can also be used depending on which geodesic distances between pixels separate the feature distances between non-matching points, and can be based on the difference between each geodesic distance in the set of third geodesic distances and each corresponding feature distance in the set of fourth feature distances. However, because the “third loss value” of Method 600 will be based on different sets of “third geodesic distances” and “fourth feature distances” as described above regarding steps 604 and 606, it will differ from the previous method. Figure 5 The "third loss value" described in Method 500. Similarly, the third loss value in Method 600 can be determined based on any suitable equation. For example, the third loss value could be the selected point p for the first image. r Each point p in the second image t (From a total of n points) Calculate the cross-view geodesic loss value L according to Equation 7 below. cd .

[0055]

[0056] Similarly, the third loss value can be based on more than one selected point p in the first image. r Therefore, the third loss value can be based on the following values, vectors, or tensors: the loss value L calculated using the set of fourth feature distances and the set of third geodesic distances according to Equation 7. cd And for other randomly selected points p r Calculated additional loss value L cd For example, the third loss value could be for a predetermined number of selected points p—e.g., 1, 5, 16, 128, etc. r Calculated loss value L cd The average value. Similarly, the third loss value can be obtained by connecting the selected points p for a predetermined number of points. r Calculated loss value L cd To determine.

[0057] In step 610, the processing system uses the set of loss values ​​obtained as described above to perform step 422 of method 400b. In this case, the set of loss values ​​includes the first and second loss values ​​determined in steps 410 and 420 of FIG. 4, and the third loss value determined in step 608. Again, this set of loss values ​​can be used in any suitable manner to influence modifications to one or more parameters of the neural network. In this regard, the processing system can sum the first, second, and third loss values ​​to obtain a total loss value and modify the parameters of the prediction model based on this total loss value. Similarly, the first, second, and third loss values ​​can be multiplied by one or more pre-selected weighting factors (e.g., a specific w) before being combined to form the total loss value. c w s and w cd value).

[0058] Figure 7 The determination of additional loss values ​​is depicted for use with Figure 4 and Figure 5 An exemplary method 700 combines the loss values ​​of methods 400a, 400b, and 500. In this respect, method 700 produces a "fourth loss value," which, together with... Figure 6 The loss value determined in method 600 is exactly the same, but it will be combined with the loss values ​​calculated in methods 400a, 400b and 500. Therefore, in step 702, the processing system executes steps 402-420 of methods 400a and 400b as described above, and steps 504-508 of method 500.

[0059] In step 704, the processing system determines the set of fifth feature distances between the seventh point represented in the first feature map and all points in the second image represented in the second feature map. Here, the seventh point can also be any randomly selected reference point p in the first image. r In some respects, the processing system can be configured to select a reference point p that is visible in the first image but not in the second image. r For example, when the first and second images depict human objects walking, the reference point p... r This can correspond to a point on the object's left leg, which is visible in the first image but is covered by the object's right leg in the second image and therefore invisible there. Again, the processing system is configured to select a reference point p that is not visible in the second image. r This allows neural networks to learn how points in a second image can correspond to points visible only in the first image (e.g., due to differences in object pose, camera position, lighting, etc., between the two images). Similarly, the label p... rIt is used for convenience and does not indicate that the selected reference point must be the same as the reference point used in step 412 of method 400b or step 504 of method 500.

[0060] If each point in the first image is represented as a vector in the first feature map, and each point in the second image is represented as a vector in the second feature map, then the set of fifth feature distances can be determined as described above with respect to Equation 2. In this regard, the set of fifth feature distances will include distances for points represented by p. r and p t The vector calculation of each point p in the second image t Individual feature distance d(p) r ,p t Similarly, as mentioned above, the set of fifth feature distances can be determined using any other suitable equation, such as by calculating the dot product of each vector pair, by calculating the cross product of each vector pair, by adding each vector pair, by subtracting each vector pair, etc. In some respects, the equation used to calculate the set of fifth feature distances may differ from the equation used to calculate the first, second, third, and / or fourth feature distances.

[0061] In step 706, the processing system determines a set of fourth geodesic distances between the first point represented in the second surface map and all points in the second image represented in the second surface map. Here again, the second surface map corresponds to the second image, which has been generated using a 3D model of the object in the same pose used to generate the second image. Furthermore, the first point in the second surface map and the seventh point in the first feature map correspond to the same features on the object's 3D model. For example, as described above, the first point in the second surface map could correspond to a point on the object's left leg that is visible in the first image and therefore represented in the first feature map, but not visible in the second image and therefore not represented in the second feature map. Therefore, using the exemplary point indicated in the description of step 704, the first point represented in the second surface map will correspond to point p represented in the first image. r Correspondingly, in this case, the set of fourth geodesic distances will include each point p for the second image. t The individual geodesic distance g(corr(p) r ),p t ), which represents the distance along the 3D surface of the second surface map between the following points: and point p of the first image. r (Here it is called corr(p)) r The corresponding points and the points p in the second image t Corresponding points.

[0062] In step 708, the processing system determines a fourth loss value from the set of loss values, which is based on the set of the fifth feature distances and the set of the fourth geodesic distances. The fourth loss value of method 600 can be determined according to any suitable equation. For example, the fourth loss value can be based on the selected point p of the first image according to equation 7 above. r Each point p in the second image t The cross-view geodesic loss value L calculated from a total of n points. cd .

[0063] Similarly, the fourth loss value can be based on more than one selected point p in the first image. r Therefore, the fourth loss value can be based on the following values, vectors, or tensors: the loss value L calculated according to Equation 7 using the set of fifth feature distances and the set of fourth geodesic distances. cd And for other randomly selected points p r Calculated additional loss value L cd For example, the fourth loss value could be a selected point p for a predetermined number—e.g., 1, 5, 16, 128, etc. r Calculated loss value L d The average value. Similarly, the fourth loss value can be obtained by connecting the selected points p for a predetermined number of times. r Calculated loss value L cd To determine.

[0064] In step 710, the processing system uses the set of loss values ​​obtained as described above to perform step 422 of method 400b. In this case, the set of loss values ​​includes the first and second loss values ​​determined in steps 410 and 420 of FIG4, in... Figure 5 The third loss value determined in step 508 and the fourth loss value determined in step 708. Again, this set of loss values ​​can be used in any suitable manner to influence the modification of one or more parameters of the neural network. In this regard, the processing system can sum the first, second, third, and fourth loss values ​​to obtain a total loss value, and modify the parameters of the prediction model based on this total loss value. Similarly, the first, second, third, and fourth loss values ​​can be multiplied by one or more pre-selected weighting factors (e.g., a specific w) before being combined to form the total loss value. c w s w d and w cd value).

[0065] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the above features can be utilized without departing from the scope defined by the claims, the foregoing description of exemplary systems and methods should be understood as exemplary rather than limiting the scope defined by the claims. Furthermore, the examples described herein, as well as the use of clauses with wording such as “such as,” “comprising,” “including,” etc., should not be construed as limiting the scope of the claims to the specific examples; rather, these examples are intended to illustrate only some of the many possible embodiments. Moreover, the same reference numerals in different figures can identify the same or similar elements.

Claims

1. A method for training a neural network to predict correspondences in two or more images, the method comprising: One or more processors of the processing system use the neural network to generate a first feature map based on a first image of the object, and to generate a second feature map based on a second image of the object, wherein the first image and the second image are different and have been generated using a three-dimensional model of the object; The one or more processors determine a first feature distance between a first point represented in the first feature map and a second point represented in the second feature map, wherein the first point and the second point correspond to the same feature on the three-dimensional model of the object; The second feature distance between the third point represented in the first feature map and the fourth point represented in the first feature map is determined by the one or more processors; The first geodesic distance between the third point and the fourth point, represented in the first surface map, is determined by the one or more processors. The first surface map corresponds to the first image and has been generated using the three-dimensional model of the object. The third feature distance between the third point represented in the first feature map and the fifth point represented in the first feature map is determined by the one or more processors. The one or more processors determine a second geodesic distance between the third point and the fifth point represented in the first surface map; A first loss value is determined from a set of loss values ​​by the one or more processors, the first loss value being based on the first feature distance; A second loss value is determined from the set of loss values ​​by the one or more processors, the second loss value being based on the second feature distance, the third feature distance, the first geodesic distance, and the second geodesic distance; and One or more parameters of the neural network are modified by the one or more processors, at least in part, based on the set of loss values.

2. The method according to claim 1, wherein, The first loss value is further based on a set of additional feature distances, each of which is a feature distance between a selected point represented in the first feature map and a corresponding point represented in the second feature map, the selected point and the corresponding point corresponding to the same feature on the three-dimensional model of the object.

3. The method according to claim 2, wherein, The first point and each selected point together represent all pixels in the first image.

4. The method according to claim 1, wherein, The second loss value is further based on at least one additional feature distance pair and at least one additional geodesic distance pair. Each given additional feature distance pair in the at least one additional feature distance pair includes two feature distances between the set of three selected points represented in the first feature map, and Each given additional geodesic distance pair in the at least one additional geodesic distance pair includes two geodesic distances between the set of three selected points represented in the first surface map.

5. The method according to claim 1, further comprising: The set of fourth feature distances between the sixth point represented in the first feature map and all other points of the first image represented in the first feature map is determined by the one or more processors; The one or more processors determine a set of third geodesic distances between the sixth point represented in the first surface map and all other points in the first image represented in the first surface map; as well as A third loss value is determined from the set of loss values ​​by the one or more processors, the third loss value being based on the set of the fourth feature distances and the set of the third geodesic distances.

6. The method according to claim 5, wherein, The third loss value is further based on at least one additional set of feature distances and at least one additional set of geodesic distances. Each given additional set of feature distances in at least one additional set of the feature distances is a set of feature distances between the selected point represented in the first feature map and all other points of the first image represented in the first feature map, and Each given additional set of geodesic distances in at least one additional set of geodesic distances is a set of geodesic distances between the selected point represented in the first surface map and all other points of the first image represented in the first surface map.

7. The method of claim 1, further comprising: The set of fifth feature distances between the sixth point represented in the first feature map and all points of the second image represented in the second feature map is determined by the one or more processors; The processors determine a set of fourth geodesic distances between a first point represented in the second surface map and all points in the second image represented in the second surface map, the second surface map corresponding to the second image and generated using the three-dimensional model of the object, and the first point in the second surface map and the sixth point in the first feature map corresponding to the same feature on the three-dimensional model of the object; as well as A third loss value is determined from the set of loss values ​​by the one or more processors, the third loss value being based on the set of the fifth feature distances and the set of the fourth geodesic distances.

8. The method according to claim 7, wherein, The third loss value is further based on at least one additional set of feature distances and at least one additional set of geodesic distances. Each given additional set of feature distances in at least one additional set of the feature distances is a set of feature distances between the selected point represented in the first feature map and all points of the second image represented in the second feature map, and Each given additional set of geodesic distances in at least one additional set of geodesic distances is a set of third geodesic distances between a corresponding point represented in the second surface map and all points in the second image represented in the second surface map, wherein the corresponding point in the second surface map and the selected point in the first feature map correspond to the same feature on the three-dimensional model of the object.

9. The method of claim 5, further comprising: The set of sixth feature distances between the seventh point represented in the first feature map and all points of the second image represented in the second feature map is determined by the one or more processors; The processors determine a set of fifth geodesic distances between a first point represented in the second surface map and all points in the second image represented in the second surface map, the second surface map corresponding to the second image and generated using the three-dimensional model of the object, and the first point in the second surface map and the seventh point in the first feature map corresponding to the same feature on the three-dimensional model of the object; as well as A fourth loss value is determined from the set of loss values ​​by the one or more processors, the fourth loss value being based on the set of the sixth feature distances and the set of the fifth geodesic distances.

10. The method according to claim 7, wherein, The first point represented in the second surface diagram corresponds to a feature on the three-dimensional model of the object that is not represented in the second feature diagram.

11. The method according to claim 1, wherein the method further comprises: The one or more processors generate at least one of the first image, the second image, or the first surface image.

12. The method according to claim 7, further comprising: The one or more processors generate at least one of the first image, the second image, the first surface map, or the second surface map.

13. The method according to claim 9, wherein the method further comprises: The one or more processors generate at least one of the first image, the second image, the first surface map, or the second surface map.

14. The method according to claim 1, wherein, The object is a human being or a representation of a human being.

15. The method according to claim 14, wherein, The object is in a different pose in the first image than it is in the second image.

16. The method of claim 14, wherein, The first image is generated from a different perspective of the three-dimensional model of the object compared to the second image.

17. A processing system, comprising: The memory stores the neural network; as well as One or more processors are coupled to the memory and configured to use the neural network to predict correspondences in two or more images. The neural network has been trained according to a training method to predict correspondences in two or more images, the training method including: A first feature map is generated based on a first image of the object, and a second feature map is generated based on a second image of the object, wherein the first image and the second image are different and have been generated using a three-dimensional model of the object; Determine a first feature distance between a first point represented in the first feature map and a second point represented in the second feature map, wherein the first point and the second point correspond to the same feature on the three-dimensional model of the object; Determine the second feature distance between the third point represented in the first feature map and the fourth point represented in the first feature map; A first geodesic distance is determined between the third point and the fourth point represented in the first surface map, which corresponds to the first image and has been generated using the three-dimensional model of the object; Determine the third feature distance between the third point represented in the first feature map and the fifth point represented in the first feature map; Determine the second geodesic distance between the third point and the fifth point represented in the first surface diagram; Determine a first loss value from the set of loss values, the first loss value being based on the first feature distance; Determine a second loss value from the set of loss values, the second loss value being based on the second feature distance, the third feature distance, the first geodesic distance, and the second geodesic distance; and One or more parameters of the neural network are modified, at least in part, based on the set of loss values.

18. The processing system according to claim 17, wherein, The neural network has been trained according to a training method to predict correspondences in two or more images, the training method further comprising: Determine the set of fourth feature distances between the sixth point represented in the first feature map and all other points of the first image represented in the first feature map; Determine the set of third geodesic distances between the sixth point represented in the first surface map and all other points in the first image represented in the first surface map; and A third loss value is determined from the set of loss values, the third loss value being based on the set of the fourth feature distances and the set of the third geodesic distances.

19. The processing system according to claim 17, wherein, The neural network has been trained according to a training method to predict correspondences in two or more images, the training method further comprising: Determine the set of fourth feature distances between the sixth point represented in the first feature map and all points of the second image represented in the second feature map; A set of third geodesic distances is determined between a first point represented in the second surface map and all points in the second image represented in the second surface map, the second surface map corresponding to the second image and generated using the three-dimensional model of the object, and the first point in the second surface map and the sixth point in the first feature map corresponding to the same feature on the three-dimensional model of the object; and A third loss value is determined from the set of loss values, the third loss value being based on the set of the fourth feature distances and the set of the third geodesic distances.

20. The processing system according to claim 18, wherein, The neural network has been trained to predict correspondences in two or more images based on a training method, the training method further comprising: Determine the set of fifth feature distances between the seventh point represented in the first feature map and all points of the second image represented in the second feature map; Determine the set of fourth geodesic distances between a first point represented in the second surface map and all points in the second image represented in the second surface map, the second surface map corresponding to the second image and generated using the three-dimensional model of the object, and the first point in the second surface map and the seventh point in the first feature map corresponding to the same feature on the three-dimensional model of the object; and A fourth loss value is determined from the set of loss values, the fourth loss value being based on the set of the fifth feature distances and the set of the fourth geodesic distances.

Images