Training a differentiable renderer and neural network for querying 3D model databases

Abstract

A system and method for a differentiable network trainable to learn an optimized query of a 3D model database for object recognition includes: training a first differentiable network configured as a differentiable renderer by generating a 2D image from a 3D model of a first object of a dissimilar second object while optimizing rendering parameters for producing the 2D image by gradient descent of a first triple loss function. Visual variation between the images is maximized. Training a second differentiable network configured as a convolutional neural network defined by a regression function by generating a searchable feature vector of the 2D image. Determining the feature vector using optimized neural network parameters determined by gradient descent of a second triple loss function to achieve high correlation with an input image of the first object and low correlation with an image of the second object.

Description

Technical Field

[0001] The present application relates to machine learning applied to 3D virtual models to support design and digital twins. More particularly, the present application relates to training differentiable renderers and neural networks for improved 3D model queries. Background Art

[0002] In industrial applications (e.g., design, manufacturing, digital twins), objects or system entities are modeled as 3D virtual representations (e.g., CAD models) to analyze 3D aspects of the objects (e.g., engineering analysis, simulation and testing, digital twins, product life management, design of new objects / systems, automated manufacturing, system performance, etc.). Access to such models requires queries against large database files stored in database repositories, the complexity, number, and size of which have proliferated in recent years, thereby stretching the limits of effective query and processing capabilities of computer systems. To illustrate the magnitude of the workload required, a 3D model of a complex object is represented as a long list of (e.g., thousands of) 3D points and vectors representing the surface of the object in 3D space. It can therefore be appreciated how complex the task of comparing a given 3D model to known 3D models in a database archive in seeking an appropriate match would be, even without considering the added complexity of different viewpoints (rotations) of the object pairing.

[0003] Conventional approaches therefore rely on visual analysis of 2D translations from 3D models when making comparisons. The task of querying 3D models based on visual similarity remains challenging because it requires the ability to decompose 3D data into relevant 2D views and appropriately compare these views. Existing solutions address this problem by generating a fixed set of views from the queried 3D model and then comparing the features of these views with a database of features extracted from images of known 3D models rendered from the same viewpoint. Convolutional neural networks (CNNs) can be trained to extract meaningful and discriminative features from the views. Feature vectors representing known images can then be stored together with their labels (e.g., the class of the object they represent) so that the CNN can extract corresponding vectors of new images from the queried object and compare them with the stored ones for retrieval. The effectiveness of this approach depends on the choice of predefined viewpoints and parameters (e.g., lighting) used to render the 2D views of the 3D CAD modeled object. In practice, the viewpoints of the rendered 2D views are often chosen arbitrarily, for example by sampling equidistant positions in the 3D space around the target object. The selection does not take into account the characteristics of the target object (e.g., for objects that exhibit multiple symmetries, the system may be tricked into choosing the wrong symmetry in the selection).

[0004] Some solutions apply 3D to 3D comparison without intermediate 2D rendering for comparison. This approach represents 3D models as meshes of vertices (i.e., their surfaces are decomposed into a large number of triangles in 3D space). However, such representations cannot be easily compared because meshes of similar objects can have different resolutions, mesh element shapes, and / or different numbers of mesh elements and vertices. Summary of the invention

[0005] Methods and systems are disclosed for a pipeline of differentiable networks that are trained to learn optimized database queries for retrieving 3D models for object recognition. A particular improvement involves training a differentiable network to automatically extract key features from a 3D model of a target object that can be used to query a complex database of virtual 3D models representing known objects and entities with a strong likelihood of matching known 3D models indexed by vision-based features. Rather than relying on manually defined rendering parameters for training a network to render a target object and extract features, the pipeline is designed for a comprehensive trainable solution that automatically regresses optimal rendering parameters (e.g., viewpoint) to render the 3D model as a 2D rendering while also optimizing network parameters to extract distinctive features from the rendered image. Once the pipeline is trained, a database of 3D models of known objects can be indexed by the trained pipeline according to distinctive features, enabling the pipeline to perform future object recognition operations on the target object through efficient querying and retrieval of the best matching 3D model.

[0006] In one aspect, a system of differentiable networks can be trained to learn an optimized database query for retrieving a 3D model used for object recognition. The system includes an end-to-end differentiable pipeline, the pipeline including a first differentiable network and a second differentiable network, the first differentiable network being configured as a differentiable renderer, the differentiable renderer being trained to generate a first set of 2D images from a first 3D model of a first object and a second set of 2D images from a second 3D model of a second object. The first set of 2D images are rendered according to optimized rendering parameters for generating the 2D images, the optimized rendering parameters being determined by gradient descent of an optimized first triplet loss function to (i) maximize visual variation between N images generated for the first 3D model and (ii) maximize visual difference between N images generated for the first model and N images generated for the second model. The second differentiable network is configured as a convolutional neural network defined by a regression function and is trained to generate searchable features of the first set of 2D images, the features being configured as feature vectors extracted from the first set of 2D images according to optimized neural network parameters, the optimized neural network parameters being determined by gradient descent of an optimized second triple loss function to achieve high correlation with input images of the first set of 2D images and low correlation with images of the second set of 2D images.

[0007] In one aspect, a method for training a pipeline of differentiable networks to learn an optimized database query for retrieving a 3D model used for object recognition includes: training a first differentiable network configured as a differentiable renderer by generating a first set of 2D images from a first 3D model of a first object and generating a second set of 2D images from a second 3D model of a second object, the first set of 2D images being rendered according to optimized rendering parameters for generating the 2D images, the rendering parameters being determined by gradient descent of an optimized first triplet loss function to (i) maximize visual variation between N images generated for the first 3D model and (ii) maximize visual difference between the N images generated for the first model and the N images generated for the second model. The method also includes training a second differentiable network configured as a convolutional neural network defined by a regression function by generating searchable features of the first set of 2D images, the features being configured as feature vectors extracted from the first set of 2D images according to the optimized neural network parameters, the optimized neural network parameters being determined by gradient descent of an optimized second triplet loss function to achieve high correlation with input images of the first set of 2D images and low correlation with images of the second set of 2D images. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like elements throughout the various figures unless otherwise specified.

[0009] Figure 1 An example of a system for object recognition using differentiable rendering to enhance 3D model retrieval according to an embodiment of the present disclosure is shown.

[0010] Figure 2 An example of a differentiable network pipeline training operation for extracting a differentiable feature vector from a 3D model according to an embodiment of the present disclosure is shown.

[0011] Figure 3 Example viewpoint sampling is shown to reduce uncertainty when rendering a target object according to an embodiment of the present disclosure.

[0012] Figure 4 An example of a computing environment is shown in which embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0013] Accessing 3D models stored in databases has become a more common task in many industrial environments. The following examples give some specific examples of this. Designers and engineers of new objects and systems can use CAD software to draft and define new objects, and may need to access existing CAD models as templates for components / parts (for example, re-using existing / standard valves, screws, etc. to design new motors). Existing CAD models of target objects can also be used for virtual testing of system designs. In automated manufacturing environments, 3D models are also increasingly used during the manufacture of the objects themselves, such as for input to 3D printers or to provide robots with the information they need to manipulate the manufactured objects and their components (for example, most algorithms that teach robots how to grasp specific objects rely on 3D models to train them).

[0014] When access to a 3D model database is required, there are situations where a 3D model of the target object is available and can be used to query the database to find the best match. Below are a few examples of such queries. In an industrial setting where an engineer or operator has a faulty part or component that needs to be replaced (e.g., a spare part is needed), but the part is not recognizable, such a part can be scanned by a 3D scanner to obtain a rough 3D model of the object (e.g., using a smartphone scanning application). The model can then be used to query the database of tagged 3D models in order to identify it. In other cases, a low-quality CAD model or 3D scan (e.g., a noisy scan) is available for the target object, and a better quality 3D model with a more accurate and detailed version is available in the 3D model database, which can be queried to search for and match to the low-quality model. Comparison of 3D models may also occur to identify duplicate or counterfeit 3D models in the database. For example, 3D models representing exactly the same object may have completely different structures (e.g., different sampling of surface 3D points); therefore, a solution that compares models based on their appearance rather than matching based on exact points would be useful.

[0015] Methods and systems are disclosed for significant improvements in the efficiency and accuracy of query and retrieval of 3D models in an archival database using a novel feature decomposition of 3D renderings through a differentiable renderer, and feature extraction through a neural network, to enable indexed searches of the database to find the best classification matches. The disclosed framework addresses the technical problem of intermediate 2D renderings with uncertainty about the appropriate viewpoints for objects with characteristic symmetries, which results in poorly matched 3D models being retrieved from the archive. Instead of the traditional two-step approach consisting of first rendering 2D views of a 3D model and then processing these images into distinctive feature vectors for recognition, an end-to-end differentiable pipeline is trained to directly use the 3D model of the object to extract a set of distinctive feature vectors. Based on the 3D model of the object, the differentiable renderer generates a set of images representing the object from various key viewpoints and using various key rendering parameters. The renderer optimizes the rendering parameters and the distinctive viewpoints to obtain a set of images that best capture the geometry of the object. A convolutional neural network (CNN) is jointly trained by performing regression to determine a set of extracted feature vectors from the rendered images. These vectors are stored in a database of indexed feature vectors. After the training pipeline, object recognition can be performed on the target object by extracting its feature vector and retrieving 3D models visually similar to the target object from the database based on clustered feature comparison over the entire set of vectors.

[0016] Figure 1 An example of a system for object recognition using differentiable rendering to enhance 3D model retrieval according to an embodiment of the present disclosure is shown. In one embodiment, a computer vision system includes a computing device 110 and a data repository 150 of 3D models interconnected by a network 130. The data repository can be configured as a database that stores a virtual representation of a long list of (e.g., thousands) points and vectors configured to represent the surface of the object in 3D space for each 3D model of the object. The computing device 110 includes a processor 115 and a memory 111 (e.g., a non-transitory computer-readable medium) on which various computer applications, modules, or executable programs are stored. The CNN module 120 and the differentiable renderer module 122 are trained together as a differentiable network pipeline that extracts features from the 3D model. The query module 124 operates on the features extracted from the trained pipeline and matches the features with a feature-based index of the data repository linked to the 3D model to predict the classification of the target object.

[0017] The user can use GUI 116 to select the 3D model of the target object to be identified, thereby inputting the selection as a path or uploading. Then, the 3D model of the target object can be sent to the differentiable renderer 122 or the applicable cloud-based differentiable renderer 142. In one embodiment, the differentiable renderer is implemented locally as a renderer 122. In another embodiment, the implementation uses a cloud-based renderer 142. In another embodiment, the renderer module 122 is implemented as a local client connected to a cloud-based renderer 142. In one embodiment, the local CNN module 120 is an independent CNN module that operates locally on the computing device 110. In another embodiment, the CNN is deployed as a cloud-based CNN 140. In another embodiment, the CNN module 120 is a local client for interfacing with a cloud-based CNN 140 for training and post-training operations. Here, the differentiable network formed by the differentiable renderer and CNN will be described as a local deployment of the renderer 122 and CNN120, however, the described configuration and operation are also applicable to alternative embodiments involving the renderer 142 and CNN 140. According to the disclosed embodiments, training of the network formed by the differentiable renderer 122 and the CNN 120 enables more efficient and relevant retrieval of strongly matching 3D models for identifying target objects.

[0018] A network 130 (e.g., a local area network (LAN), a wide area network (WAN), or an Internet-based network) connects the computing device 110 to a data repository of 3D models 150 that will be indexed by the trained differentiable network for improved search and retrieval of 3D models during object recognition operations.

[0019] In one aspect of the disclosed embodiment, a triplet loss function may be derived to train the differentiable pipeline formed by the differentiable renderer 122 and the CNN 120. According to the method, the CNN 120 is trained to extract relevant features from the image x and encode them into a low-dimensional feature vector in an embedding space of low dimensionality d. in, (e.g., mapping the image to a descriptor space where object instances are well separated). Then we can represent the known image x s The eigenvector z s With its label y s The vectors are stored together (i.e., object classes for object retrieval tasks) so that given a new image, CNN 120 can be used to extract its corresponding vector and compare it with the stored vectors for recognition.

[0020] Therefore, in order to learn the discriminative features, a triplet loss function is applied to the training of the CNN 120, and the CNN applies a regression function, which is written as: where θT Represents the trainable parameters of CNN. The triplet is defined as (x b , x + , x - ), where x b From the training dataset The input image is used to represent a specific category c b and the viewpoint (e.g., represented by the quaternion q b Definition) binding anchor, x + is the same as x in terms of labels b Similar positive images (similar categories c + =c b and / or gesture q + ≈q b ), x - are negative samples with different contents (different categories c - and posture q - ). Here it is represented as The triplet loss function is designed to force the network to extract b and x + Similar features and b and x - Dissimilar features:

[0021]

[0022] in

[0023] Here, ∈ is a margin that sets the minimum ratio of distances between similar and dissimilar pairs of samples. Once trained, the CNN regression function Used to calculate These vectors are then used as keywords to index the model labels and form a database of feature descriptors linked to a data repository of known 3D models 150. Recognition of unknown objects is accomplished by using the trained network to compute feature descriptors for each rendered 3D model and then finding its nearest feature descriptor in the database as an indicator of the best prediction for classification by applying a nearest neighbor search algorithm.

[0024] Figure 2An example of a differentiable network pipeline training operation for extracting differentiable feature vectors from a 3D model according to an embodiment of the present disclosure is shown. The differentiable pipeline 200 includes a first network G for a differentiable renderer 122 and a second network T that is a neural network of CNN 120. In one embodiment, the pipeline 200 is trained using a pair of different models, an image 230 of each model in the pair is rendered, and a feature vector Z is extracted from the image according to a triple loss function that learns the parameters of the differentiable networks G and T. Once trained, all archived 3D models of the database 150 are run through the pipeline 200 to generate an index of the entire database, where each model is indexed according to the feature vector associated with each model.

[0025] In one embodiment, Figure 2 As shown in the training flowchart in FIG. 1 , the pipeline 200 receives 3D models 210 for rendering by the differentiable renderer 122 and returns a set of N rendered images 230 for each 3D model. Figure 2 The simplified example shown is a first model m A Render two images x A1 ,x A2 (N=2), these two images have different poses P A1 and P A2 Similarly, the second model m from a different class B Two images x B1 ,x B2 Also use two different postures P B1 and P B2 According to an embodiment of the present disclosure, the setting of N can be extended to capture more postures (for example, N=3 to 100). The functionality of the differentiable renderer 122 can be expressed as a function: It takes as input a 3D model m and returns a set of rendered images 230, such as from various key viewpoints and with various key rendering parameters θ G (lighting conditions, material properties, etc.) indicates category c m N images of the object In one embodiment, the differentiable renderer 122 includes a differentiable network (i.e., composed entirely of differentiable operations) that updates the rendering parameters θ by back-propagating gradient descent. G An advantage of forming a pipeline with a differentiable renderer 122 and a CNN 120 (differentiable end-to-end) is that both can be trained together as a single trainable solution for optimizing the parameters (or weights) of the combined differentiable pipeline using gradient descent. The CNN 120 can be represented by the regression function: Indicates that it obtains the rendered image 230, for example, a set of N images X m, and returns a set of features 250, such as N feature vectors One advantage of the feature vector representation is that the unique characteristics of the object are captured (i.e., the extracted features have a high correlation with the input image and a low correlation with images of different objects), which enables improved search and matching of the system at runtime. Another advantage of processing feature vectors is the compressed dimensionality of the data representing the object. For example, an object may have a matrix representation of size 300×400, while the feature vector may be only 1×100. Note that CNN 120 can extract feature vectors image by image, or it can extract all feature vectors at once, making it possible to aggregate and transfer information between different vectors (e.g., batch normalization, attention mechanisms, recursive architectures, etc.).

[0026] Training pipeline 200, so that given a new 3D model m r , extract relevant features Z mr , and then compared with a feature database of known 3D models to search for the most similar features. In this pipeline training, the purpose of the rendering module 122 is to extract a 2D view of a highly distinguishable 3D object. For example, Figure 3 The importance of smart viewpoint sampling for view-based 3D object retrieval is illustrated. Assume that two rendered views are used to represent a given 3D object. As shown in Sample A, object 310 is rendered as image 313 from viewpoint 311, and image 322 from viewpoint 312. Here, the viewpoints are poorly sampled (i.e., both viewpoints 311 and 312 are located on a common axis of object 310, and the resulting images 313, 314 leave room for uncertainty (e.g., incorrectly assuming from both images 313, 314 that they represent a spherical object). Using a smart sampling approach, such as Sample B, where viewpoints 311 and 322 are more diverse than 311 and 312, the result is a more explicit "shape description" in the form of image pairs 313 and 324. The same reasoning about parameters that affect the appearance of an object in a set of images can be applied to other rendering parameters, including but not limited to lighting conditions, material properties, camera parameters, etc. Therefore, the renderer 122 is trained to optimize its rendering parameters θ G A set of images X (i.e., N different viewpoints, N lighting conditions, etc.) is selected to obtain a set of images X that best captures the geometric and visual properties of the object (e.g., selecting viewpoints that are not oriented toward a common axis of symmetry in the 3D dataset, selecting various light conditions to cast different shadows and highlight different surface features, etc.).

[0027] Back to Figure 2, we now describe differentiable end-to-end training for a generalized example where N sample images are rendered. In each training iteration, two different objects are randomly selected from the pool of known and modeled objects from the repository 150, and their respective 3D models m associated with object A and object B are trained. A and m B is fed to the differentiable renderer 122. Given its rendering parameters θ G The renderer 122 generates two sets of N images X A and X B ,in:

[0028] and (for their respective objects)

[0029] The two sets of images are passed to CNN 120, which outputs feature vectors Z respectively. A and Z B ,in:

[0030] And Z B Defined similarly.

[0031] The loss function of the CNN 120 output layer calculates the loss value, which is then back-propagated through the pipeline to update the parameter θ according to the following gradient descent calculation T and θ G The triplet loss function of CNN 120 is derived by extending equation (1) to formulate a triplet loss function that depends on both the renderer 122 and the CNN 120, has two sets of input images, and is applied to the training of the CNN 120 The minimum loss is sought for the following function:

[0032]

[0033] where ∈ = 2arccos|q A ·q B | A viewpoint with a quaternion from θ G Apply equation (2) to Figure 2 In the setting, anchor image 231 corresponds to Similar model images corresponds to image x_A1, and different images Corresponding to image x_B1 or x_B2. Loss function The purpose of is to maximize the similarity between feature vectors of all views of the same object (high correlation) while maximizing the dissimilarity with all feature vectors from different objects (low correlation). In one embodiment, the model m is inverted by calculating the second loss value A and m B The effect of performing backpropagation on each of these triplet loss values ​​can make the loss more symmetric. For example, in Equation (2) if the model m A and m B Inversion, not image As the anchor image for comparison, X B One of the images in the group may represent the anchor. Computing the combination of the two loss values ​​makes the training more robust because it constrains the network 200 to immediately only apply the first model m A The feature vector and the second model m B For additional robustness, additional loss values ​​can be computed and back-propagated by allowing each of the four images to be used as an anchor, such as Figure 2 In the case shown, four loss values ​​are given for N=2.

[0034] Using loss function CNN 120 is trained to look at visual variations and map images from different views of the same model to similar feature vectors, while also ensuring that the feature vectors of images from different models are well pushed apart. The task of CNN 120 is to learn the difference between “images from different views of the same model” and “images from different models”, returning similar descriptors in the former case and different descriptors in the latter case. The loss function of CNN 120 is The training of is different from the training of renderer 122 so that CNN 120 understands what each model looks like and that different images actually contain similar features.

[0035] Different loss functions are applied to function G of the renderer 122, with the goal of rendering highly dissimilar images for different objects, but also as dissimilar as possible between images of the same object (e.g., using viewpoint differences). In one aspect, the roles of the renderer 122 and CNN 120 are opposing, but complementary. The set of rendered images captures the overall appearance of the model, not by representing the model from the same viewpoint, but from a meaningful selection of viewpoints that most highlight the variation in the model's appearance. The parameters of the renderer 122 (e.g., viewpoint list, lighting conditions, etc.) should be optimized to maximize the visual variation within the set of images rendered from each object, thereby best capturing the overall appearance of the object rather than just a specific appearance, such as from one side.

[0036] The loss function of the renderer 122 is calculated according to the following relationship

[0037]

[0038] in

[0039] and

[0040] Defining functions To optimize G to maximize its benefits from each model (e.g., to achieve Figure 3 The visual variation (i.e., the most dissimilar) among the N images generated by sampling B) is defined. To optimize the function G to maximize the visual difference between the N images of model A and the N images of model B. The triple loss function of Equation 3 Relying on the regression function of CNN 120 (Equations 3a, 3b) Since a strong distance metric is needed to compare images, this is actually The purpose of being trained is that it is optimized to extract semantically rich features from images, getting rid of the limitations of pixel values. Instead of directly calculating the distance between images, the renderer 122 can generate images that are semantically different rather than just pixel-wise different. Function used to calculate the renderer 122 When the loss is Freeze at the same time The parameters (i.e., The freezing parameters of are not affected by the result of this loss calculation).

[0041] Embedding 250 represents the feature vector Z derived by CNN 120 for each of the four rendered images 230 in an embedding space (lower dimensional space) Figure 2 251, 252 is represented as a space having three dimensions (for illustration purposes, the feature vectors in this example have three dimensions), so each vector can be represented as a point in 3D space. The size ratio indicates the number of features or characteristics represented. However, for real-world applications, the number of such features can be much greater than 3. The feature vectors 251, 252 are similar to the model m of the target model. A The rendered image is associated with the feature vector 253 and the model m B, which is different from the target model. As shown, the training goal is to derive network parameters that pull similar feature vectors closer together in the embedding space while pushing feature vectors of different models further apart. For example, a feature vector can represent an anchor vector, such as feature vector 251 associated with anchor image 231 of four image inputs 230, to which other vectors are compared. The pull / push distance is a result of the triple loss function described above. The triple loss function is formulated to pull feature vectors of semantically similar images 251, 252 together (i.e., this forces CNN 120 to extract feature vectors that are close to each other to minimize the denominator in the triple loss equation). The triple loss formula also pushes feature vectors 253 of different samples away from the anchor feature vector 251. Backpropagating this triple loss in the pipeline essentially teaches the network parameters to pull features of similar content together and push features of different content apart.

[0042] After the training phase of pipeline 200, known 3D models are fed through the trained pipeline, and for each known 3D model, a feature vector is extracted and then used as a key to generate a model label (e.g., m ) with a feature descriptor. A 、m B ) and forms a feature descriptor database linked to a data repository of known 3D models 150.

[0043] In the inference-based prediction process, the trained pipeline 200 is used to retrieve 3D models that are visually similar to a given 3D model from a known database. As before, the new 3D model may involve a new unknown object, where the 3D model can be obtained from a 3D scan of the new object. Another instance of a new 3D model that requires a database query may include a rough 3D model that needs to be replaced by a more detailed and accurate model. Given a new 3D model, the trained pipeline is used to obtain a set of feature vectors Z of the new 3D model corresponding to N images rendered for the model. Each vector in Z can be compared with a feature descriptor database linked to a data repository of known 3D models 150. Prediction can be achieved using a query module 124 by aggregating results over the entire set (e.g., by counting the most present object classes in the nearest neighbors), and the query module 124 identifies the most similar 3D model in the database. For example, for each of the N feature vectors, the query module 124 queries the database with the feature vector to find the nearest neighbor, retrieves the nearest neighbor and the corresponding label (e.g., model ID, such as m A ). The query module 124 may aggregate all results from the database to return the most similar model ID as a prediction for the classification, such as the model that appears most frequently in the results of the N queries.

[0044] Figure 4An example of a computing environment in which embodiments of the present disclosure may be implemented is shown. Computing environment 400 includes a computer system 410, which may include a communication mechanism such as a system bus 421 or other communication mechanism for transmitting information within the computer system 410. The computer system 410 also includes one or more processors 420 coupled to the system bus 421 for processing information. In one embodiment, computing environment 400 corresponds to a system for performing the above-described embodiments, wherein computer system 410 relates to a computer described in more detail below.

[0045] Processor 420 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other processors known in the art. More generally, the processor described herein is a device for executing machine-readable instructions stored on a computer-readable medium for performing tasks, and may include any one or a combination of hardware and firmware. The processor may also include a memory storing machine-readable instructions executable for performing tasks. The processor acts on information by manipulating, analyzing, modifying, converting, or transmitting information used by an executable program or information device and / or by routing the information to an output device. The processor may use or include the capabilities of, for example, a computer, a controller, or a microprocessor, and may be adjusted using executable instructions to perform special functions that are not performed by a general-purpose computer. The processor may include any type of suitable processing unit, including but not limited to a central processing unit, a microprocessor, a reduced instruction set computer (RISC) microprocessor, a complex instruction set computer (CISC) microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a digital signal processor (DSP), etc. In addition, processor 420 can have any suitable micro-architecture design, and it comprises any number of components, such as register, multiplexer, arithmetic logic unit, cache controller, branch predictor etc. for controlling the read / write operation to cache memory.The micro-architecture design of processor can support any instruction set in multiple instruction sets.Processor can be connected (electrically connected and / or comprise executable component) with any other processor that can interact and / or communicate therebetween.User interface processor or maker are known elements, comprise electronic circuit or software or both combinations for generating display image or its part.User interface comprises one or more display images that enable user to interact with processor or other equipment.

[0046] The system bus 421 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may allow information exchange (e.g., data (including computer executable code), signaling, etc.) between various components of the computer system 410. The system bus 421 may include, but is not limited to, a memory bus or a memory controller, a peripheral bus, an accelerated graphics port, etc. The system bus 421 may be associated with any suitable bus architecture, including, but not limited to, an Industry Standard Architecture (ISA), a Micro Channel Architecture (MCA), an Enhanced ISA (EISA), a Video Electronics Standards Association (VESA) architecture, an Accelerated Graphics Port (AGP) architecture, a Peripheral Component Interconnect (PCI) architecture, a PCI Express architecture, a Personal Computer Memory Card International Association (PCMCIA) architecture, a Universal Serial Bus (USB) architecture, etc.

[0047] Continue to refer Figure 4 , the computer system 410 may also include a system memory 430 coupled to the system bus 421 for storing information and instructions to be executed by the processor 420. The system memory 430 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as a read-only memory (ROM) 431 and / or a random access memory (RAM) 432. RAM 432 may include other dynamic storage devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). ROM 431 may include other static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 430 may be used to store temporary variables or other intermediate information during the execution of instructions by the processor 420. The basic input / output system 433 (BIOS) contains basic routines that help transfer information between elements within the computer system 410, such as during startup, and may be stored in ROM 431. RAM 432 may contain data and / or program modules that are immediately accessible to and / or currently being operated by the processor 420. The system memory 430 may also include, for example, an operating system 434, an application program module 435, and other program modules 436. The application program module 435 may include a program for Figure 1 The above modules described may also include a user portal for developing applications, allowing parameters to be entered and modified if necessary.

[0048] An operating system 434 may be loaded into memory 430 and may provide an interface between other application software executing on computer system 410 and the hardware resources of computer system 410. More specifically, operating system 434 may include a set of computer executable instructions for managing the hardware resources of computer system 410 and providing common services to other applications (e.g., managing memory allocations between various applications). In certain example embodiments, operating system 434 may control the execution of one or more program modules depicted as being stored in data store 440. Operating system 434 may include any operating system now known or that may be developed in the future, including but not limited to any server operating system, any host operating system, or any other proprietary or non-proprietary operating system.

[0049] The computer system 410 may also include a disk / media controller 443 coupled to the system bus 421 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 441 and / or a removable media drive 442 (e.g., a floppy disk drive, an optical drive, a tape drive, a flash drive, and / or a solid-state drive). The storage device 440 may be added to the computer system 410 using an appropriate device interface. (e.g., a small computer system interface (SCSI), an integrated device electronics (IDE), a universal serial bus (USB), or a firewire). The storage devices 441, 442 may be external to the computer system 410.

[0050] The computer system 410 may include a user input / output interface module 460 to process user input from a user input device 461, which may include one or more devices such as a keyboard, a touch screen, an input tablet, and / or a pointing device, for interacting with a computer user and providing information to the processor 420. The user interface module 460 also processes system output to a user display device 462 (e.g., via an interactive GUI display).

[0051] The computer system 410 may perform part or all of the processing steps of the disclosed embodiments in response to the processor 420 executing one or more sequences of one or more instructions contained in a memory such as the system memory 430. Such instructions may be read into the system memory 430 from another computer-readable medium of the memory 440 (e.g., a magnetic hard disk 441 or a removable media drive 442). The magnetic hard disk 441 and / or the removable media drive 442 may contain one or more data stores and data files used by the disclosed embodiments. The data store 440 may include, but is not limited to, a database (e.g., relational, object-oriented, etc.), a file system, a flat file, a distributed data store in which data is stored on more than one node of a computer network, a peer-to-peer network data store, etc. The data store contents and data files may be encrypted to improve security. The processor 420 may also be used in a multi-processing device to execute one or more instruction sequences contained in the system memory 430. In alternative embodiments, hard-wired circuits may be used in place of or in conjunction with software instructions. Therefore, the embodiments are not limited to any particular combination of hardware circuits and software.

[0052] As above, the computer system 410 may include at least one computer-readable medium or memory for storing instructions programmed according to embodiments of the present disclosure, and for containing data structures, tables, records, or other data described herein. The term "computer-readable medium" used herein refers to any medium that participates in providing instructions to the processor 420 for execution. Computer-readable media can take many forms, including but not limited to non-temporary, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid-state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disks 441 or removable media drives 442. Non-limiting examples of volatile media include dynamic memory, such as system memory 430. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute the system bus 421. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0053] The computer-readable medium instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional process programming languages ​​such as "C" programming language or similar programming languages. Computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing the state information of computer-readable program instructions to perform various aspects of the present disclosure.

[0054] Various aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable medium instructions.

[0055] The computing environment 400 may also include a computer system 410 operating in a networked environment using logical connections to one or more remote computers, such as a remote computing device 473. The network interface 470 may enable communication with other remote devices 473 or systems and / or storage devices 441, 442, for example, via a network 471. The remote computing device 473 may be a personal computer (notebook or desktop), a mobile device, a server, a router, a network PC, a peer device, or other public network node, and typically includes many or all of the elements described above with respect to the computer system 410. When used in a networked environment, the computer system 410 may include a modem 472 for establishing communications over a network 471, such as the Internet. The modem 472 may be connected to the system bus 421 via the user network interface 470 or via another appropriate mechanism.

[0056] The network 471 can be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or a series of connections, a cellular telephone network, or any other network or medium that can facilitate the communication between the computer system 410 and other computers (e.g., a remote computing device 473). The network 471 can be wired, wireless, or a combination thereof. The wired connection can be implemented using Ethernet, a universal serial bus (USB), RJ-6, or any other wired connection known in the art. The wireless connection can be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellites, or any other wireless connection methods known in the art. In addition, several networks can work alone or communicate with each other to facilitate the communication in the network 471.

[0057] It should be understood that Figure 4 The program modules, applications, computer executable instructions, codes, etc. described as being stored in system memory 430 are merely illustrative and not exhaustive, and the processing described as being supported by any particular module may alternatively be distributed across multiple modules or performed by different modules. In addition, various program modules, scripts, plug-ins, application programming interfaces (APIs), or any other suitable computer executable code hosted locally on computer system 410, remote devices 473, and / or hosted on other computing devices accessible via one or more networks 471 may be provided to support the processing by Figure 4 The functions and / or additional or alternative functions provided by the program modules, applications or computer executable codes depicted in the Figure 4 The processing collectively supported by the set of program modules depicted in the description may be performed by a fewer or greater number of modules, or the functionality described as supported by any particular module may be supported at least in part by another module. In addition, the program modules supporting the functionality described herein may form part of one or more applications that may be executed on any number of systems or devices according to any suitable computing model, such as a client-server model, a peer-to-peer model, or the like. In addition, the processing collectively supported by the set of program modules depicted in the description may be performed by a fewer or greater number of modules, or the functionality described as supported by any particular module may be supported at least in part by another module. In addition, the program modules supporting the functionality described herein may form part of one or more applications that may be executed on any number of systems or devices according to any suitable computing model, such as a client-server model, a peer-to-peer model, or the like. Figure 4 Any functionality supported by any program module depicted in the description may be implemented, at least in part, in hardware and / or firmware on any number of devices.

[0058] It should also be understood that the computer system 410 may include alternative and / or additional hardware, software or firmware components other than those described or depicted without departing from the scope of the present disclosure. More specifically, it should be understood that the software, firmware or hardware components depicted as forming part of the computer system 410 are merely illustrative, and that certain components may not be present or additional components may be provided in various embodiments. Although various illustrative program modules have been depicted and described as software modules stored in the system memory 430, it should be understood that the functionality described as supported by the program modules may be enabled by any combination of hardware, software and / or firmware. It should be further understood that in various embodiments, each of the above modules may represent a logical partitioning of the supported functions. The logical partitioning is depicted for ease of explanation of the functions and may not represent the structure of the software, hardware and / or firmware used to implement the functions. Therefore, it should be understood that in various embodiments, the functionality described as provided by a particular module may be provided at least in part by one or more other modules. In addition, in some embodiments, one or more of the depicted modules may not be present, while in other embodiments, additional modules that are not depicted may be present and may support at least a portion of the functionality and / or additional functionality. Furthermore, while certain modules may be depicted and described as sub-modules of another module, in certain embodiments such modules may be provided as stand-alone modules or sub-modules of other modules.

[0059] Although specific embodiments of the present disclosure have been described, those of ordinary skill in the art will recognize that there are many other modifications and alternative embodiments within the scope of the present disclosure. For example, any function and / or processing capability described about a specific device or component can be performed by any other device or component. In addition, although various illustrative implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will appreciate that many other modifications to the illustrative implementations and architectures described herein are also within the scope of the present disclosure. In addition, it should be understood that any operation described herein as being based on another operation, element, component, data, etc., elements, components, data, etc., can be additionally based on one or more other operations, elements, components, data, etc. Therefore, the phrase "based on" or its variants should be interpreted as "based at least in part on".

[0060] The flow chart and block diagram in the figure show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure.In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing a specified logic function.In some replaceable embodiments, the function indicated in the square frame may not occur in the order indicated in the accompanying drawings.For example, according to the function involved, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order.It will also be noted that each frame in the block diagram and / or the flow chart illustration and the combination of frames in the block diagram and / or the flow chart illustration can be realized by a system based on dedicated hardware that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

Claims

1. A system for a differentiable network trainable to learn optimized queries of a 3D model database for object recognition, the system comprising: processor; as well as a non-transitory memory having stored thereon modules executed by the processor, the modules comprising: a database of virtual representation data of a plurality of 3D models of objects, the data being configured as a list of points and vectors representing surfaces of the objects in 3D space; An end-to-end differentiable pipeline, comprising a first differentiable network and a second differentiable network, wherein the first differentiable network is configured as a differentiable renderer, the differentiable renderer is trained to generate a first set of 2D images from a first 3D model of a first object and a second set of 2D images from a second 3D model of a second object, the first set of 2D images being rendered according to optimized rendering parameters for generating 2D images, the rendering parameters determined by gradient descent of an optimized first triple loss function maximizing visual variation between N images generated for the first 3D model and maximizing visual difference between N images generated for the first model and N images generated for the second model; and Wherein, the second differentiable network is configured as a convolutional neural network, the convolutional neural network is defined by a regression function and is trained to generate searchable features of the first set of 2D images, the features are configured as feature vectors extracted from the first set of 2D images according to optimized neural network parameters, the optimized neural network parameters are determined by gradient descent of an optimized second triple loss function to achieve high correlation with input images of the first set of 2D images and low correlation with images of the second set of 2D images.

2. The system according to claim 1, wherein: The first triplet loss function relies on a calculated distance between the feature vectors extracted by the second triplet loss function.

3. The system according to claim 1, wherein: The first triplet loss function is optimized to capture the overall appearance of the first object and the second object by representing meaningfully selected viewpoints in terms of appearance variations.

4. The system according to claim 1 further comprises a data repository of known 3D models, wherein: After training, the trained pipeline is used to receive each known 3D model, determine a feature vector associated with each known 3D model, and generate an index of model labels with feature descriptors for the known 3D models to form a feature descriptor database linked to a data repository of the known 3D models.

5. The system according to claim 4, wherein: The trained pipeline is configured to receive a new 3D model of an unknown target object and generate a feature vector for the new 3D model; The system further comprises: The query module is configured as follows: comparing each of the feature vectors with the feature descriptors of the feature descriptor database; Find the nearest neighbors of a feature descriptor, Retrieve nearest neighbors and corresponding model labels, and Returning a prediction of a classification of the unknown target object corresponding to the model label of one of the nearest neighbors.

6. A method for training a pipeline of differentiable networks to learn optimized queries of a 3D model database for object recognition, the method comprising: training a first differentiable network configured as a differentiable renderer by generating a first set of 2D images from a first 3D model of a first object and a second set of 2D images from a second 3D model of a second object, the first set of 2D images being rendered according to optimized rendering parameters for generating 2D images, the rendering parameters determined by gradient descent of an optimized first triplet loss function maximizing visual variation between N images generated for the first 3D model and maximizing visual difference between the N images generated for the first model and the N images generated for the second model; as well as A second differentiable network configured as a convolutional neural network defined by a regression function is trained by generating searchable features of the first set of 2D images, the features being configured as feature vectors extracted from the first set of 2D images according to optimized neural network parameters, the optimized neural network parameters being determined by gradient descent of an optimized second triplet loss function to achieve high correlation with input images of the first set of 2D images and low correlation with images of the second set of 2D images.

7. The method according to claim 6, wherein: The first triplet loss function relies on a calculated distance between the feature vectors extracted by the second triplet loss function.

8. The method according to claim 6, wherein: The first triplet loss function is optimized to capture the overall appearance of the first object and the second object by representing meaningfully selected viewpoints in terms of appearance variations.

9. The method according to claim 6, further comprising: After training, the trained pipeline is used to receive each known 3D model, determine a feature vector associated with each known 3D model, and generate an index of model labels with feature descriptors for the known 3D models to form a feature descriptor database linked to a data repository of the known 3D models.

10. The method according to claim 9, further comprising: Receiving a new 3D model of an unknown target object through the trained pipeline; Generating a feature vector of the new 3D model; comparing each feature vector with the feature descriptor of the feature descriptor database; Find the nearest neighbors of feature descriptors; Retrieve nearest neighbors and corresponding model labels; as well as A prediction of a classification of the unknown target object corresponding to the model label of one of the nearest neighbors is returned.

Images