Accelerated training of machine learning models based on NeRF

Abstract

Systems, methods, and non-transitory computer-readable media for obtaining a set of content items for training a neural-radiance field (NeRF)-based machine learning model for object recognition are disclosed. Therein, a depth map of an object depicted in the set of content items can be determined, and a first set of training data comprising a reconstructed content item depicting only the object can be generated from the depth map. Further, a second set of training data comprising one or more optimal training paths associated with the set of content items can also be generated from the depth map. The one or more optimal training paths are generated at least in part from a dissimilarity matrix associated with the set of content items. The NeRF-based machine learning model can be trained from the first set of training data and the second set of training data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method and system for accelerating training of a machine learning model based on NeRF, and to a non-transitory memory realized based on the foregoing method. BACKGROUND

[0002] Deep learning-based machine learning techniques have achieved numerous successes in facial recognition, detection, and segmentation techniques. Recently, techniques using neural radiance fields (NeRF) for surface reconstruction have gained attention in facial recognition. In a neural radiance field, volumetric rendering of an object in three-dimensional space is modeled, and the volumetric density of the object is used as a weight to train a neural network used in facial recognition. Compared to traditional facial recognition techniques, a NeRF-based machine learning model (e.g., a neural network) can reconstruct a surface that is smoother, more continuous, and has a higher spatial resolution. In some cases, a NeRF-based machine learning model can use less computational storage space compared to traditional techniques. While NeRF-based machine learning models have numerous advantages over traditional techniques in facial recognition, training such machine learning models can be laborious and time-consuming. For example, a NeRF-based machine learning model for facial recognition can require weeks of training. SUMMARY

[0003] In various embodiments of the present disclosure, systems, methods, and non-transitory computer-readable media for obtaining a set of content items for training a neural radiance field (NeRF)-based machine learning model for object recognition are described. In some embodiments, a depth map of an object depicted in the set of content items can be determined, and a first set of training data including reconstructed content items that depict only the object can be generated based on the depth map. In addition, a second set of training data including one or more optimal training paths associated with the set of content items can be generated based on the depth map. The one or more optimal training paths can be generated based at least in part on a dissimilarity matrix associated with the set of content items. The NeRF-based machine learning model can be trained based on the first set of training data and the second set of training data.

[0004] In some embodiments, the depth map of the object depicted in the set of content items can be determined by computing intrinsic and extrinsic parameters of a camera that captured the set of content items. Based on the intrinsic and extrinsic parameters, a coarse point cloud associated with the object depicted in the set of content items can be determined. Based on the coarse point cloud, a mesh of the object depicted in the set of content items can be determined. Based on the mesh of the object, the depth map of the object depicted in the set of content items can be determined.

[0005] In some embodiments, the intrinsic and extrinsic parameters of the cameras can be determined by Structure from Motion (SfM) techniques, and the mesh of the object can be determined by Poisson Reconstruction techniques.

[0006] In some embodiments, the intrinsic and extrinsic parameters of the cameras and the mesh of the object are determined by Multi-View Depth Fusion techniques.

[0007] In some embodiments, the first set of training data can be determined by a way of determining pixels in each content item in the set of content items to be filtered out. Then, the pixels in each content item in the set of content items can be filtered out. Further, the remaining pixels in each content item in the set of content items can be sampled to generate the reconstructed content item.

[0008] In some embodiments, the pixels in each content item in the set of content items to be filtered out can be determined by a way of determining pixels in each content item in the set of content items that are outside of a depth threshold range indicated by a respective depth map of each content item. The depth threshold range can indicate a depth range of at least one object depicted in each content item.

[0009] In some embodiments, the second set of training data can be generated by a way of determining depth map matching metric values of the set of content items. Further, contour matching metric values of the set of content items can also be determined. From the depth map matching metric values and the contour matching metric values, a dissimilarity matrix associated with the set of content items can be generated. From the dissimilarity matrix, a connectivity graph associated with the set of content items can be generated. By applying a minimum spanning tree technique to the connectivity graph, the one or more optimal training paths associated with the set of content items can be generated. The minimum spanning tree technique can rearrange the connectivity graph into a plurality of sub-trees, and each path of the plurality of sub-trees can be an optimal training path.

[0010] In some embodiments, the depth map matching metric values of the set of content items can be determined by a way of comparing depth maps of two content items in the set of content items. The two content items can depict one object. Further, dissimilarity values of each depth point in the depth maps of the two content items can be calculated. In addition, the dissimilarity values of depth points in the depth maps of the two content items can be added to generate a depth map matching metric value of the two content items.

[0011] In some embodiments, the contour matching measure value for the object can be determined by comparing depth maps of two content items among the set of content items. The two content items can depict an object. Further, contour information associated with the object contained in the depth maps of the two content items can be compared. Based on the comparison of the contour information, a contour matching measure value for the two content items can be calculated.

[0012] In some embodiments, columns and rows of the dissimilarity matrix can correspond to frame numbers associated with the set of content items. Values of the dissimilarity matrix can indicate dissimilarity between any two content items among the set of content items indicated by corresponding frame numbers. The values of the dissimilarity matrix can be determined based on corresponding depth map matching measure values and the contour matching measure values of any two content items among the set of content items.

[0013] The above and other features of the apparatuses, systems, methods, and non-transitory computer-readable media disclosed herein, the manner in which the BRIEF DESCRIPTION OF DRAWINGS BRIEF DESCRIPTION OF DRAWINGS

[0014] The specific features of various embodiments of the present technology are set forth in the claims that follow. The features and advantages of the present technology will become better understood from the following detailed description, accompanying drawings, and appended claims taken in conjunction with the accompanying drawings, in which:

[0015] Figure 1 An example system including an object recognition module that recognizes objects is shown in accordance with various embodiments disclosed herein;

[0016] Figure 2 An example training data preparation module is shown in accordance with various embodiments disclosed herein;

[0017] Figure 3A Example reconstructed content items depicting objects and example depth ranges are shown in accordance with various embodiments disclosed herein;

[0018] Figure 3B A method of generating reconstructed content items that depict only an object of interest is shown in accordance with various embodiments disclosed herein, wherein the reconstructed content items are used to train a NeRF-based machine learning model for object recognition;

[0019] Figure 3CA flowchart showing generation of one or more optimal training paths for training a NeRF-based machine learning model for object recognition according to various embodiments of the present disclosure is shown;

[0020] Figure 4 A method of training a NeRF-based machine learning model for object recognition according to various embodiments of the present disclosure is shown.

[0021] Figure 5 A block diagram of a computer system that can be used to implement any of the various embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Deep learning-based machine learning techniques have achieved numerous successes in facial recognition, detection, and segmentation techniques. Recently, techniques using neural radiance fields (NeRF) for surface reconstruction have gained attention in facial recognition. In a neural radiance field, a volumetric rendering of an object in three-dimensional space is modeled, and the volumetric density of the object is used as a weight to train a neural network used in facial recognition. Compared to traditional facial recognition techniques, a NeRF-based machine learning model (e.g., a neural network) can reconstruct a surface that is smoother, more continuous, and has a higher spatial resolution. In some cases, a NeRF-based machine learning model can use less computational storage space compared to traditional techniques. While a NeRF-based machine learning model has numerous advantages over traditional techniques in facial recognition, the training of such a machine learning model can be laborious and time-consuming. For example, a NeRF-based machine learning model for facial recognition can require weeks of training. Thus, a NeRF-based machine learning model can not be suitable for commercial applications.

[0023] The present disclosure describes a solution to the above problem. In various embodiments, a machine learning model, such as a multi-layer perceptron (MLP) neural network, can be trained to recognize features (or facial features) of an object (or a human face) based on a neural radiance field associated with the object. As described above, object recognition (or facial recognition) based on a trained NeRF-based machine learning model has numerous advantages over traditional object recognition techniques. However, the training of such a machine learning model can be time-consuming. Thus, to reduce the time required for training a NeRF-based machine learning model, the training data used for training a NeRF-based machine learning model can be preprocessed. By preprocessing the training data, the time required for training a NeRF-based machine learning model can be reduced. In the present disclosure, “object recognition” and “facial recognition” can be used interchangeably. The techniques described in the present disclosure can be used for object recognition and / or facial recognition applications.

[0024] In various embodiments, the training data used for training the NeRF-based machine learning model for object recognition can include a set of content items (e.g., images, videos, cyclic videos, etc.). The set of content items can depict various objects and / or object features. In some embodiments, the set of content items can be pre-processed to determine object depth maps depicted in the set of content items. For example, for an image depicting a person in a scene, the distance between the camera that took the image and the person can be estimated. In this example, the distances to various points on the person (e.g., the head, the body, etc.) can be estimated first, and then a depth map of the person can be generated using such distances. A depth map generally contains information about the depths (e.g., distances) of the surfaces of an object depicted in a content item at a perspective associated with the content item. An object depth map can be determined based on an object mesh (e.g., a geometric shape or a polygonal representation) of the object. The object mesh can be determined based on a coarse point cloud of the object depicted in the set of content items. The coarse point cloud of the object can be computed based on the intrinsic and extrinsic parameters of the camera used to take the set of content items. After the object depth map is determined, two sets of training data for training the NeRF-based machine learning model for object recognition can be generated.

[0025] In some embodiments, the first set of the two sets of training data can include reconstructed content items. The reconstructed content items can be generated from the set of content items based on the object depth maps. For example, for an image depicting a person, the depth map of the person can be superimposed on the image. In this example, by superimposing the depth map on the image, the depth (e.g., distance) of the person at the perspective of the image can be determined. After the depth of the person is determined, only the image pixels corresponding to the person are sampled to construct a reconstructed image that depicts only the person. In this example, other pixels of the image are discarded or not sampled. In this way, the size (e.g., file size) of the training data can be greatly reduced. Furthermore, since the reconstructed content items, instead of the normal content items, are used for training, the time required for training the NeRF-based machine learning model can also be reduced. For example, for an image that depicts a person in the foreground and trees in the background, the object of interest is the person. By sampling only the pixels in the reconstructed image that correspond to the person, the training of the NeRF-based machine learning model for object recognition only considers the person and not the trees. In this way, the training of the NeRF-based machine learning model can only train the object (in this case, the person) that the NeRF-based machine learning model is to recognize.

[0026] In some embodiments, the second group of the two groups of training data can include one or more optimal training paths for the NeRF-based machine learning model. The one or more optimal training paths can enable parallel training of the NeRF-based machine learning model, thereby accelerating the training of the NeRF-based machine learning model. In some embodiments, each of the one or more optimal training paths can include one or more content items that depict the same object in sequence (e.g., in time sequence, in action sequence, etc.) or in different perspectives. In some embodiments, the one or more optimal training paths can be generated according to a complete graph corresponding to the group of content items of the training data. The complete graph can be constructed according to a dissimilarity matrix associated with the group of content items. In the present disclosure, the dissimilarity matrix generally represents the dissimilarity between any two content items (e.g., images or image frames) of the group of content items that depict the same or similar objects. The dissimilarity matrix can accelerate the multi-frame training of the NeRF-based machine learning model by identifying or grouping content items that depict the same or similar objects in sequence or in different perspectives. In some embodiments, the values of the dissimilarity matrix can be determined according to depth map matching metric values and contour matching metric values of the group of content items. The depth map matching metric values can be determined by comparing the depth maps of any two content items that depict the same or similar objects in sequence or in different perspectives. The contour matching metric values can be determined by comparing the contours of the same or similar objects contained in the depth maps of any two content items that depict the objects in sequence or in different perspectives. After the complete graph is constructed, the complete graph can be evaluated by a minimum spanning tree technique to generate the one or more optimal training paths, where the values of the dissimilarity matrix are the edge weights in the minimum spanning tree technique. The minimum spanning tree technique can arrange the group of content items in a manner that minimizes the dissimilarity between the objects depicted in the group of content items in the training paths. In this way, the training of the NeRF-based machine learning model can be optimized, thereby reducing the time required for the training. The above and other features of the present solution will be further described in detail below.

[0027] Figure 1An example system 100 according to various embodiments of the present application is shown, which includes an object recognition module 110 for recognizing objects. In various embodiments, the object recognition module 110 can be implemented as a NeRF-based machine learning model trained to recognize objects depicted in content items (e.g., images, videos, looping videos, etc.) through volumetric rendering of the objects. The objects depicted in the content items can include, for example, human faces, facial features, animals, vehicle types, vehicle license plates, etc. The NeRF-based machine learning model can be implemented through any suitable machine learning technique. For example, the NeRF-based machine learning model can be implemented through a multi-layer perceptron (MLP) neural network. In some cases, the NeRF-based machine learning model can be implemented through one or more logistic regression-based classifiers. Among other implementations, there are many other implementations. In some embodiments, the object recognition module 110 can be implemented in part or in whole as software, hardware, or any combination thereof. In some embodiments, the object recognition module 110 can be implemented in part or in whole as software that allows execution in one or more computing devices or systems, such as a cloud computing system. For example, a trained NeRF-based machine learning model can be implemented in part or in whole in a cloud computing system to recognize objects or object features depicted in a feed of captured images or videos. Among other implementations, there are many other implementations.

[0028] In some embodiments, as Figure 1As shown, the system 100 can further include at least one data store 120. The object recognition module 110 can be configured to communicate with and / or operate in conjunction with the at least one data store 120. The at least one data store 120 can store various types of data associated with the object recognition module 110. For example, the at least one data store 120 can store training data used for training a NeRF-based machine learning model for object recognition. The training data may, for example, include images, videos, and / or cyclical videos depicting various objects. For example, the at least one data store 120 can store multiple images depicting a cat to train a NeRF-based machine learning model to recognize a cat. In some embodiments, the at least one data store 120 can store various intrinsic and extrinsic parameters of a camera, coarse point clouds, depth maps, etc. that can be accessed by the object recognition module 110. In some embodiments, the at least one data store 120 can store various metric values and dissimilarity metric values that can be accessed by the object recognition module 110. In some embodiments, the at least one data store 120 can store machine-readable instructions (e.g., code) that, when executed, cause one or more computing systems to train a NeRF-based machine learning model for object recognition or to identify a recognition training object for a NeRF-based machine learning model. In some embodiments, the at least one data store 120 can include a database storing information related to human faces. For example, the at least one data store 120 can include a database storing features of human faces. This database can be used to identify a human recognized by a trained NeRF-based machine learning model. For example, a face recognized by a trained NeRF-based machine learning model can be compared to a database storing features of faces of criminals or suspects.

[0029] In some embodiments, the object recognition module 110 can include a training data preparation module 112 and a machine learning training module 114. The training data preparation module 112 is configured to pre-process training data used for training of a NeRF-based machine learning model for object recognition. By pre-processing the training data, the time required for training of the NeRF-based machine learning model can be reduced. In some embodiments, the training data preparation module 112 can obtain a set of content items for training of the NeRF-based machine learning model. The set of content items can include, for example, images, videos, looping videos depicting various objects. For example, a set of images depicting various facial features can be utilized as training data for training of a NeRF-based neural network to recognize faces and compare the recognized faces with information stored in the at least one data storage 120. In some embodiments, the training data preparation module 112 can determine depth maps of objects depicted in the set of content items. A depth map generally contains information about the depth (e.g., distance) of surfaces of an object depicted in a content item at a perspective associated with the content item. Based on the object depth maps, the training data preparation module 112 can generate a first set of training data containing reconstructed content items depicting only the objects and a second set of training data containing one or more optimal training paths to be used for training of the NeRF-based machine learning model. The training data preparation module 112 is described in further detail below in connection with Figure 2

[0030] In some embodiments, the machine learning training module 114 can be configured to train the NeRF-based machine learning model for object recognition. The machine learning training module 114 can train the NeRF-based machine learning model based on the first and second sets of training data generated by the training data preparation module 112. The machine learning training module 114 can train the NeRF-based machine learning model in parallel based on the reconstructed content items contained in the first set of training data and the one or more optimal training paths contained in the second set of training data. For example, a reconstructed image depicting only facial features can be used as input training data for a NeRF-based MLP neural network and one or more optimal image training paths can be used as weights for the NeRF-based MLP neural network to simultaneously train the NeRF-based MLP neural network to recognize human faces after training. In this way, the time required for training of the NeRF-based MLP neural network can be reduced. As described above, conventional training methods for the NeRF-based machine learning model can be extremely time-consuming. By pre-processing the training data used for training of the NeRF-based machine learning model, the time required for training can be reduced by several orders of magnitude.

[0031] ​Figure 2 An example training data preparation module 200 according to various embodiments of the present disclosure is shown. In some embodiments, Figure 1 The training data preparation module 112 can be implemented as the training data preparation module 200. As Figure 2 As shown, in some embodiments, the training data preparation module 200 can include a depth map determination module 202, an object reconstruction module 204, and a content item sequence generation module 206. Each of the modules is described in detail below.

[0032] In some embodiments, the depth map determination module 202 can be configured to determine a depth map of an object depicted in a training data content item. As described above, a depth map generally contains information related to the depth (e.g., distance) of a viewpoint associated with a surface of an object depicted in a content item. For example, for an image depicting a person in a scene, the depth map determination module 202 can determine the depth (e.g., distance) of the person relative to a viewpoint of the scene at each depth point (e.g., head, torso, etc.) associated with the person. In some embodiments, the depth map determination module 202 can determine a depth map of an object depicted in a content item by first computing the intrinsic and extrinsic parameters of a camera used to capture the content item. The intrinsic (or internal) parameters of a camera can include, for example, the focal length and lens distortion of the camera. The extrinsic (or external) parameters of a camera can include, for example, parameters that describe the transformation between the camera and its external environment. For example, the extrinsic parameters can include a rotation matrix used to rotate or translate an object depicted in a content item. In some embodiments, the depth map determination module 202 can utilize Structure from Motion (SfM) techniques to determine the intrinsic and extrinsic parameters of a camera. Structure from Motion is a photogrammetry technique used to determine the spatial and geometric shape of an object depicted in a content item through the movement of a camera. In some cases, the depth map determination module 202 can utilize multi-view depth fusion techniques to determine the intrinsic and extrinsic parameters of a camera. Many other implementations exist.

[0033] In some embodiments, the depth map determination module 202 can generate a coarse point cloud of an object depicted in a content item based on the intrinsic and extrinsic parameters of a camera. The coarse point cloud of an object can represent the shape and / or contour of the object as a three-dimensional surface in a three-dimensional space. For example, the intrinsic or extrinsic parameters of a camera used to capture an image depicting a person's face can be estimated. In this example, the depth map determination module 202 can generate a coarse point cloud of the face based on the intrinsic or extrinsic parameters. In this coarse point cloud, the facial features of the face are represented as a three-dimensional surface having various local peaks and valleys that highlight the contour of the face (e.g., facial features).

[0034] In some implementations, the depth map determination module 202 can generate meshes of the objects depicted in the content items from the coarse point clouds. A mesh is generally a polygonal shape (e.g., triangle, square, rectangle, etc.) in a three-dimensional space that represents the shape and / or contours of an object represented by the coarse point cloud. For example, the depth map determination module 202 can generate a mesh of a face from a facial coarse point cloud. In this example, various contours of the face are represented by a plurality of polygonal shapes, such as triangles, to highlight various facial features of the face. In this way, the computational load required to render such surfaces can be reduced while facilitating the visualization of the surface contours. From these meshes, the depth map determination module 202 can determine a depth map of the objects depicted in the content items. The object depth in the depth map can be estimated by tracing a pixel ray to each mesh point (e.g., point of a polygonal shape) of the object. In some implementations, the depth map determination module 202 can generate the meshes of the objects from a Poisson reconstruction technique.

[0035] In some implementations, the object reconstruction module 204 can be used to sample the pixels in the content items that are required to build the objects depicted in the content items in the reconstructed content items from the training data. The sampled pixels can be used to generate the reconstructed content items, which can then be used to train the NeRF-based machine learning model for object recognition. For example, a first image can depict a person in the foreground and trees in the background. In this example, the object reconstruction module 204 can be used to sample only the pixels in the first image that correspond to the person. The sampled pixels are used to build the person in a second image, which is used to train the NeRF-based machine learning model to recognize the person. As described above, in this way, the time required to train the NeRF-based machine learning model can be reduced. In addition, the file size of the content items used to train the NeRF-based machine learning model (i.e., the reconstructed content items only depict the objects of interest) can also be reduced.

[0036] In some embodiments, the object reconstruction module 204 can identify, from the object depth map, the pixels in the content item that are needed to construct the object depicted in the content item. The object depth map can include information related to the depth (e.g., distance) of the object surface relative to the perspective of the content item. Such depth can form the basis of a depth threshold range for filtering the pixels corresponding to the object. For example, the pixels corresponding to the depth outside the depth threshold range can be discarded (filtered out or not sampled) because they are not pixels representing the object. Meanwhile, only the pixels corresponding to the depth within the depth threshold range can be sampled for constructing the object in the reconstructed content item. In this way, the object reconstruction module 204 can sample the pixels corresponding to the object depicted in the content item based on whether the pixels are within the object depth threshold range based on the object depth map. From the sampled pixels, the object reconstruction module 204 can construct the object in a set of reconstructed content items for training the NeRF-based machine learning model for object recognition. The set of reconstructed content items can be used as input (e.g., training data) for training the NeRF-based machine learning model. The object reconstruction module 204 is described in further detail below with reference to Figure 3A and Figure 3B

[0037] In some embodiments, the object reconstruction module 204 can uniformly sample the pixels corresponding to the object depicted in the content item within N uniformly spaced sampling zones and sample the pixels within the N uniformly spaced sampling zones for constructing the object in the reconstructed content item. This approach can further reduce the file size of the content item used for training the NeRF-based machine learning model. However, this approach can result in a low utilization of the sampling space, which can adversely affect the quality of the reconstructed content item. Accordingly, to minimize the problem of low utilization of the sampling space, the sampling of the pixels within the N uniformly spaced sampling zones can be dynamically adjusted. For example, the sampling of the face depicted in the reconstructed image can be adjusted from the pixel data stored in the N uniformly spaced sampling zones. In this example, the resolution of the face can not be sufficient to represent the various contours of the face. Accordingly, the sampling from the N uniformly spaced sampling zones can be adjusted by sampling more pixel data corresponding to the face for constructing the reconstructed image.

[0038] ​In some embodiments, the object reconstruction module 204 can be configured to remove noise associated with the reconstructed content item. Filtering out pixels that do not correspond to the depicted object in the content item can often result in noise in the reconstructed content item that only depicts the object. Such noise is particularly prevalent around the edges or contours of the depicted object in the reconstructed content item. Accordingly, in some embodiments, the object reconstruction module 204 can be configured to remove or minimize noise through a density supervision technique under the direction or guidance of a user. In the density supervision technique, the grid associated with the reconstructed content item needs to be monitored with human supervision to remove noise caused by non-sampled pixels (i.e., filtered out pixels). In some cases, the density supervision technique can enable accelerated training of the NeRF-based machine learning model for object recognition.

[0039] In some embodiments, the optimal content item sequence generation module 206 can be configured to generate one or more optimal training paths of the training data content items. The one or more optimal training paths can accelerate the training of the NeRF-based machine learning model for object recognition. Each of the one or more optimal training paths can include one or more content items that depict the same object in sequence (e.g., in temporal sequence, in action sequence, etc.) or at different perspectives. For example, the training data used for training the NeRF-based machine learning model for object recognition can include multiple images that depict various objects. The multiple images can be organized in a manner such that one or more images that depict the same object in the multiple images can be arranged in sequence. In some embodiments, the optimal content item sequence generation module 206 can generate the one or more optimal training paths based on a complete graph associated with the training data content items. Each node in the complete graph can correspond to a content item in the training data. In some embodiments, the complete graph can be constructed based on a dissimilarity matrix associated with the training data content items. The rows and columns of the dissimilarity matrix can represent frame numbers of the content items, and the values or dissimilarity metric values of the dissimilarity matrix can be used as edge weights for evaluating the complete graph in a minimum spanning tree technique. In the minimum spanning tree technique, the complete graph can be rearranged into multiple sub-trees based on the values of the dissimilarity matrix. Each path of the multiple sub-trees can represent one or more content items of an optimal training path.

[0040] In some embodiments, the values (e.g., dissimilarity metric values) of the dissimilarity matrix can be determined in the following manner:

[0041] F i，j = D i，j · (1 - S i，j )

[0042] where F i,j is the value (e.g., dissimilarity metric value) of the dissimilarity matrix i row (e.g., frame i of the training data content items) j column (e.g., frame j of the training data content items), Di,j For depth map matching metrics between frame i and frame j, S i,j This is the contour matching metric between frame i and frame j. The depth map matching metric is used to compare the depth map differences between two content items (such as frame i and frame j). In some implementations, the depth map matching metric between any two content items in the training data can be determined as follows:

[0043]

[0044] Where, d Fi For the frame F at viewpoint c i Depth map, d Fj For the frame F at viewpoint c j The depth map, M is the frame F i and frame F j The total number of viewpoints in the depth map. Therefore, the depth map matching metric is the sum of all depth differences between the depth maps of any two content items (e.g., frame i and frame j) depicting an object. The contour matching metric is used to compare the contours or contour information of the objects depicted by the two content items (e.g., frame i and frame j) based on their depth maps. In some implementations, the contour matching metric between any two content items in the training data can be determined as follows:

[0045]

[0046] Among them, I c i,j U represents the intersection of the contours of frames i and j from viewpoint c. c i,j Let M be the union of the contours of frames i and j from viewpoint c, and let F be the union of the contours of frames i and j from viewpoint c. i and frame F j The total number of views in the depth map. Below, combined with... Figure 3C The optimal content item sequence generation module 206 is described in further detail.

[0047] Figure 3A The diagram shows an example of reconstructed content item 300 and an example depth range 320 depicting an object according to various embodiments of this disclosure. Figure 3A As shown, the reconstructed content item 300 (e.g., an image) depicts only object 302 and not anything else (e.g., dark areas of the reconstructed content item 300). In various embodiments, the reconstructed content item 300 can be generated by sampling the pixels (e.g., "light rays") of the original content item depicting object 302. Specifically, only pixels in the original content item corresponding to object 302 (e.g., "light ray sampling areas") are sampled, while pixels not corresponding to object 302 in the original content item (e.g., "light ray discarded areas") are not sampled.

[0048] In some embodiments, each pixel in the original content item can be associated with a depth range, such as depth range 320. The depth range of each pixel can be determined from a depth map of the original content item and includes a depth threshold range, such as depth threshold range 322, that represents the depth of the object 302 depicted in the original content item that each pixel represents. Furthermore, the depth range of each pixel can be compared to the depth threshold range. If the depth range of a pixel is outside the depth threshold range, then the pixel does not represent the object 302 and is not sampled for the reconstructed content item 300. If the depth range of a pixel is within the depth threshold range, then the pixel represents the object 302 and is sampled for the reconstructed content item 300. For example, as shown in Figure 3A depth range 320 has a depth of “d”. Since this depth is outside the depth threshold range 322, this pixel corresponding to the depth range 320 is not sampled for the reconstructed content item 300.

[0049] Figure 3B depicted in the original content item. As shown in Figure 3B depicted in the original content item. As shown in

[0050] Figure 3C depicted in the original content item. As shown in Figure 3CAs shown, the processor of the computing system can obtain a set of content items (such as a "frame sequence") that sequentially depict objects for training a NeRF-based machine learning model (as shown in reference numeral 382). Based on the object depth map, the processor can construct a fully connected graph associated with the set of content items (as shown in reference numeral 384). Each node in the fully connected graph represents one content item from the set of content items. The fully connected graph can be constructed based on a dissimilarity matrix of the set of content items. This dissimilarity matrix represents the degree of dissimilarity between the objects depicted in the set of content items. The processor can evaluate the fully connected graph using a minimum spanning tree technique, thereby rearranging the fully connected graph into multiple subtrees (as shown in reference numeral 386). Each path in the multiple subtrees corresponds to a content item in the optimal training path used to train the NeRF-based machine learning model. The processor can extract one or more optimal training paths from the multiple subtrees (as shown in reference numeral 388). The processor can then use the one or more optimal training paths to train the NeRF-based machine learning model.

[0051] Figure 4 The diagram illustrates a method 400 for training a NeRF-based machine learning model for object recognition according to various embodiments disclosed herein. In this flowchart and other flowcharts, method 400 is shown as a series of blocks by way of example. It should be understood that, where applicable, each block can be understood as proceeding in parallel or in some other order. Furthermore, for clarity, some blocks that should be included may be omitted from the figures to avoid providing unnecessary information. Additionally, although some blocks could be omitted, they are still listed in the figures for illustrative purposes. The description of the other figures may also apply. Figure 4 .

[0052] In box 402, the processor (as with) Figure 1 The processor associated with the object recognition module 110 can obtain a set of content items for training a NeRF-based machine learning model. In block 404, the processor can determine a depth map of the objects depicted in the set of content items. In block 406, the processor can generate a first set of training data containing reconstructed content items that only depict the objects, based on the depth map. In block 408, the processor can generate a second set of training data containing one or more optimal training paths associated with the set of content items, based on the depth map. In block 410, the processor can train the NeRF-based machine learning model using the first and second sets of training data.

[0053] The techniques described can be implemented, for example, by one or more special-purpose computing devices. The special-purpose computing devices can be hard-wired to perform the techniques, or can include one or more special-purpose integrated circuits (ASICs) or field programmable gate arrays (FPGAs) or other programmable-logic devices that are permanently programmed to perform the techniques, or can include one or more hardware processors programmed to perform the techniques pursuant to instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices can also combine commercially available hardware computing elements with those custom-built for the

[0054] Figure 5 A block diagram of a computer system 500 is shown in FIG. 5, which can be employed as any of the various embodiments described herein. The computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a one or more hardware processors 504 coupled with bus 502 for processing information. By way of example, the computer system 500 can be a specially- engineered computer or a combination of computers for performing the functions described herein, or can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer- readable medium to be such special-purpose computer when configured and / or reconfigured.

[0055] The computer system 500 also includes a main memory 506, such as a random access memory (RAM), cache and / or other dynamic storage devices, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 also can be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions can be stored or carried on a computer-readable medium, which can be a storage media of a computer-readable medium. Such instructions, when stored in storage media accessible to processor 504, render the computer system 500 into a special-purpose machine that carries out the operations specified in the instructions.

[0056] The computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. Also coupled to bus 502 is a storage device 510, such as a magnetic disk, optical disk, USB (flash) or other storage device, for storing information and instructions.

[0057] Computer system 500 can be coupled via bus 502 to a display 512, such as a cathode ray tube (CRT) or liquid crystal display (LCD) (or a touch screen) or other display device, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, can be coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is cursor control 516, such as a mouse, a trackball or cursor direction keys for communicating direction information and command selections to processor 504 and for

[0058] Unless otherwise required by context, the term "includes" as used herein is understood to be an open term that also means "consisting of." Numeric ranges are intended to include each and every value and sub-range within the stated range. In other words, any numerical range disclosed herein is intended to include each and every value and sub-range within that range. Also, unless otherwise clear from context, when a range is disclosed, it is intended to include the end value and each intervening value. Also, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. "At least one" and "one or more" are understood to be open terms that are to be interpreted the same as "one or more." The phrases "at least one of," "at least one selected from a group of," "at least one selected from the group consisting of," and the like are to be understood to be open terms that are to be interpreted the same as "one or more of."

[0059] As used throughout the specification, "an embodiment" or "one embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Accordingly, appearances of the phrases "in one embodiment" or "in an embodiment" or "in at least one embodiment" or the like in various places throughout the specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0060] When a component is described as being operative to perform a function, that component can be understood to operate in the same or a similar manner to another component and / or to include the same or similar features, characteristics, and parameters as another component.

Claims

1. A method for training a machine learning model based on neural radiation field (NeRF) for object recognition, characterized in that, The method includes: Obtain a set of content items for training the NeRF-based machine learning model; Determine the depth map of the objects depicted in the set of content items; Based on the depth map, a first set of training data is generated, which includes only reconstructed content items depicting the object; Based on the depth map, a second set of training data is generated, including one or more optimal training paths associated with the set of content items, wherein the one or more optimal training paths are generated at least in part based on the dissimilarity matrix associated with the set of content items; and The NeRF-based machine learning model is trained based on the first set of training data and the second set of training data.

2. The method as described in claim 1, characterized in that, Determining the depth map of the object depicted in the set of content items includes: Based on the set of content items, calculate the intrinsic and extrinsic parameters of the camera that captured the set of content items; Based on the intrinsic and extrinsic parameters, determine a coarse point cloud associated with the object depicted in the set of content items; Based on the coarse point cloud, determine the mesh of the objects depicted in the set of content items; and Based on the object's grid, determine the depth map of the object depicted in the content item.

3. The method as described in claim 2, characterized in that, The intrinsic and extrinsic parameters of the camera are determined using the Structure for Motion Restoration (SfM) technique, and the mesh of the object is determined using the Poisson reconstruction technique.

4. The method as described in claim 2, characterized in that, The intrinsic and extrinsic parameters of the camera and the mesh of the object are determined by multi-view depth fusion technology.

5. The method as described in claim 1, characterized in that, Generating the first set of training data, which includes the reconstructed content items, includes: Based on the depth map, determine the pixels to be filtered out in each of the set of content items; Filter out the pixels in each of the set of content items; and The remaining pixels in each of the set of content items are sampled to generate the reconstructed content item.

6. The method as described in claim 5, characterized in that, Determining the pixels to be filtered out in each of the set of content items includes: Identify pixels in each of the set of content items that are outside the depth threshold range indicated by the corresponding depth map of each content item, wherein the depth threshold range indicates the depth range of at least one object described in each content item.

7. The method as described in claim 1, characterized in that, Generating the second set of training data, which includes one or more optimal training paths, includes: Determine the depth map matching metric for the set of content items; Determine the contour matching metric for the set of content items; Based on the depth map matching metric and the contour matching metric, generate the dissimilarity matrix associated with the set of content items; Based on the dissimilarity matrix, a connected graph associated with the set of content items is generated; and By applying the minimum spanning tree technique to the connected graph, one or more optimal training paths associated with the set of content items are generated, wherein the minimum spanning tree technique rearranges the connected graph into multiple subtrees, and each path of the multiple subtrees is an optimal training path.

8. The method as described in claim 7, characterized in that, The depth map matching metric for the set of content items is determined in the following manner: Compare the depth maps of two content items in the set of content items, the two content items depicting an object; Calculate the dissimilarity value of each depth point in the depth map of the two content items; as well as The dissimilarity values ​​of the depth points in the depth maps of the two content items are added together to generate a depth map matching metric for the two content items.

9. The method as described in claim 7, characterized in that, The contour matching metric for the set of content items is determined in the following manner: Compare the depth maps of two content items in the set of content items, the two content items depicting an object; Compare the contour information associated with the objects contained in the depth maps of the two content items; as well as Based on the comparison results of the contour information, the contour matching metric of the two content items is calculated.

10. The method as described in claim 7, characterized in that, The columns and rows of the dissimilarity matrix correspond to the frame numbers associated with the set of content items. The value of the dissimilarity matrix indicates the dissimilarity between any two content items in the set of content items indicated by the corresponding frame number. The value of the dissimilarity matrix is ​​determined based on the corresponding depth map matching metric and contour matching metric of any two content items in the set of content items.

11. A system, characterized in that, include: At least one processor; as well as A memory storing instructions, which, when executed by the at least one processor, cause the system to implement a training method for a neural radiation field (NeRF)-based machine learning model for object recognition, the method comprising: Obtain a set of content items for training the NeRF-based machine learning model; Determine the depth map of the objects depicted in the set of content items; Based on the depth map, a first set of training data is generated, which includes only reconstructed content items depicting the object; Based on the depth map, a second set of training data is generated, including one or more optimal training paths associated with the set of content items, wherein the one or more optimal training paths are generated at least in part based on the dissimilarity matrix associated with the set of content items; and The NeRF-based machine learning model is trained based on the first set of training data and the second set of training data.

12. The system as claimed in claim 11, characterized in that, Determining the depth map of the object depicted in the set of content items includes: Based on the set of content items, calculate the intrinsic and extrinsic parameters of the camera that captured the set of content items; Based on the intrinsic and extrinsic parameters, determine a coarse point cloud associated with the object depicted in the set of content items; Based on the coarse point cloud, determine the mesh of the objects depicted in the set of content items; and Based on the object's grid, determine the depth map of the object depicted in the content item.

13. The system as described in claim 11, characterized in that, Generating the first set of training data, which includes the reconstructed content items, includes: Based on the depth map, determine the pixels to be filtered out in each of the set of content items; Filter out the pixels in each of the set of content items; and The remaining pixels in each of the set of content items are sampled to generate the reconstructed content item.

14. The system as described in claim 13, characterized in that, Determining the pixels to be filtered out in each of the set of content items includes: Identify pixels in each of the set of content items that are outside the depth threshold range indicated by the corresponding depth map of each content item, wherein the depth threshold range indicates the depth range of at least one object described in each content item.

15. The system as claimed in claim 11, characterized in that, Generating the second set of training data, which includes one or more optimal training paths, includes: Determine the depth map matching metric for the set of content items; Determine the contour matching metric for the set of content items; Based on the depth map matching metric and the contour matching metric, generate the dissimilarity matrix associated with the set of content items; Based on the dissimilarity matrix, a connected graph associated with the set of content items is generated; and By applying the minimum spanning tree technique to the connected graph, one or more optimal training paths associated with the set of content items are generated, wherein the minimum spanning tree technique rearranges the connected graph into multiple subtrees, and each path of the multiple subtrees is an optimal training path.

16. A non-transitory memory of a computing system, the non-transitory memory storing instructions, characterized in that, When executed by at least one processor, this instruction causes the computing system to implement a training method for a neural radiation field (NeRF)-based machine learning model for object recognition, the method comprising: Obtain a set of content items for training the NeRF-based machine learning model; Determine the depth map of the objects depicted in the set of content items; Based on the depth map, a first set of training data is generated, which includes only reconstructed content items depicting the object; Based on the depth map, a second set of training data is generated, including one or more optimal training paths associated with the set of content items, wherein the one or more optimal training paths are generated at least in part based on the dissimilarity matrix associated with the set of content items; and The NeRF-based machine learning model is trained based on the first set of training data and the second set of training data.

17. The non-transitory memory as claimed in claim 16, characterized in that, Determining the depth map of the object depicted in the set of content items includes: Based on the set of content items, calculate the intrinsic and extrinsic parameters of the camera that captured the set of content items; Based on the intrinsic and extrinsic parameters, determine a coarse point cloud associated with the object depicted in the set of content items; Based on the coarse point cloud, determine the mesh of the objects depicted in the set of content items; and Based on the object's grid, determine the depth map of the object depicted in the content item.

18. The non-transitory memory as claimed in claim 16, characterized in that, Generating the first set of training data, which includes the reconstructed content items, includes: Based on the depth map, determine the pixels to be filtered out in each of the set of content items; Filter out the pixels in each of the set of content items; and The remaining pixels in each of the set of content items are sampled to generate the reconstructed content item.

19. The non-transitory memory as claimed in claim 18, characterized in that, Determining the pixels to be filtered out in each of the set of content items includes: Identify pixels in each of the set of content items that are outside the depth threshold range indicated by the corresponding depth map of each content item, wherein the depth threshold range indicates the depth range of at least one object described in each content item.

20. The non-transitory memory as claimed in claim 16, characterized in that, Generating the second set of training data, which includes one or more optimal training paths, includes: Determine the depth map matching metric for the set of content items; Determine the contour matching metric for the set of content items; Based on the depth map matching metric and the contour matching metric, generate the dissimilarity matrix associated with the set of content items; Based on the dissimilarity matrix, a connected graph associated with the set of content items is generated; and By applying the minimum spanning tree technique to the connected graph, one or more optimal training paths associated with the set of content items are generated, wherein the minimum spanning tree technique rearranges the connected graph into multiple subtrees, and each path of the multiple subtrees is an optimal training path.

Images