Pig single-view three-dimensional reconstruction method based on deep learning

Abstract

The invention discloses a pig single-view three-dimensional reconstruction method based on deep learning, and the method is called Magfup. In order to solve the problems that single-view three-dimensional reconstruction viewpoint learning is unstable and training is difficult due to lack of real three-dimensional data, an implicit-explicit combined three-dimensional representation method is adopted, high-dimensional data features subjected to super-resolution optimization are combined, a high-density pig three-dimensional grid can be reconstructed only through a single image, and the problem that training is difficult is solved. And data acquisition cost and equipment requirements are reduced. According to the method, the three-dimensional grid surface is optimized through the Laplacian smoothing method, and the reconstruction quality is improved. According to the invention, the model can learn the general shape and posture characteristics of the pig, predicts the three-dimensional posture, shape and texture information of the specific pig, and more accurately and comprehensively reflects the behavior characteristics of the pig. Therefore, the method can adapt to complex and changeable environments and pig behaviors in a pig farm, and a more comprehensive and accurate data basis is provided for pig behavior analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of pig three-dimensional reconstruction, and is specifically a pig single-view reconstruction method based on deep learning. Figure 3 Dimensional reconstruction method. Background Art

[0002] Animal husbandry is an important pillar industry for the development of my country's agricultural and rural economy, and pig farming occupies a pivotal position in my country's animal husbandry. Behavioral analysis of pigs not only helps to improve breeding efficiency and economic benefits, but also helps to promote animal welfare and health management. Although there has been great progress in the current research on pig behavior analysis, most of the research content focuses on identifying pig behavior from 2D image information, but this information is incomplete. Animals in real scenes are three-dimensional and have depth. Therefore, pig behavior analysis based on computer vision requires more complete information, which requires obtaining three-dimensional information of pigs. Animal three-dimensional reconstruction can be mainly divided into model-based reconstruction and model-free reconstruction. The model-based animal three-dimensional reconstruction method requires the use of a predefined model to know the reconstruction process. Although this method can provide more accurate three-dimensional reconstruction results, it requires high accuracy of the model used. However, if the model itself has defects or does not match the shape of the actual animal, it may lead to inaccurate reconstruction results. Model-free 3D reconstruction methods do not rely on predefined models and directly recover 3D structures and surfaces from input 2D images or videos, which is more consistent with the image and can better adapt to the complexity and diversity of the pig farm environment. Figure 3 For high-dimensional reconstruction, it is difficult to learn the viewpoint of an object, and the reconstructed surface is prone to missing details and cannot accurately reflect the smooth features of the object. The present invention proposes a deep learning-based 3D reconstruction method for pigs, Magfulap, which optimizes data features in high-dimensional space through a super-resolution method, which not only improves the viewpoint learning efficiency, but also advances skeleton learning and shape learning to accelerate the model convergence speed; and optimizes the 3D mesh surface through the Laplace smoothing method to improve the quality of model reconstruction. Summary of the invention

[0003] The purpose of the present invention is to provide a single-view deep learning pig Figure 3 Dimensional reconstruction method.

[0004] The technical solution adopted by the present invention is as follows: a single-view deep learning-based pig Figure 3 A dimensional reconstruction method, the method comprising the following steps:

[0005] S1: Arrange the cameras first and collect data;

[0006] S2: The data is then subjected to object detection and segmentation, and the image and mask are resized to a resolution of 256×256.

[0007] S3: Use the FeatUp framework to improve the spatial resolution of DINO features and capture subtle spatial details;

[0008] S4: Use signed distance function (SDF) to implicitly represent the pig shape, and use differentiable marching tetrahedron (DMTet) to convert SDF into explicit mesh;

[0009] S5: Use DINO-ViT to extract feature descriptors of RGB images and learn the pig skeleton structure, shape and appearance information;

[0010] S6: Generate pig surface texture map, image mask and DINO features using Lambertian illumination model and differentiable deferred mesh rendering technology;

[0011] S7: Combine reconstruction loss, view loss and regularization term to train the model;

[0012] S8: Use the cosine Laplacian matrix to smooth the mesh and reduce geometric abruptness

[0013] In a preferred embodiment, in step S1, the camera is directed toward the pig to capture RGB image data of different directions, angles and light intensities.

[0014] In a preferred embodiment, in step S2, labelimg is used to label and crop the image, eiseg is used to label the cropped image with a mask, and then the size of the cropped image and the mask is adjusted to 256×256 according to the image mask.

[0015] In a preferred embodiment, in step S3, the input image is slightly jittered, including padding, scaling and horizontal flipping operations, and the DINO model is applied to each transformed image to extract a series of low-resolution feature maps; these tiny image jitters enable subtle differences in the output features to be observed and provide sub-feature information for training the upsampler; next, a consistent high-resolution feature map is constructed from these views, which can reproduce the "jitter" of the low-resolution features after downsampling; the downsampling method is similar to the rendering step in NeRF, but the difference is that it tracks the parameters of each image used for "jittering" and applies these transformations to the learned high-resolution features before downsampling; then, the downsampled features are compared with the true model output using Gaussian likelihood loss;

[0016] More specifically, let t∈T be a set of small transformations, such as margin, scaling, cropping, horizontal flipping, and their combinations; let x be the input image, f be the model backbone, σ ↓ is the learned downsampler, σ ↑ is the learned upsampler; it can be evaluated by hr =σ ↑ (f(x),x) to form the high-resolution features for prediction; this parameterization allows σ ↑ As a guided upsampler (depends on both x and f(x)), an unguided upsampler (depends only on f(x)), an implicit network (depends only on x), or a learned feature buffer that does not depend on anything; the main multi-view reconstruction loss term can be formed as follows:

[0017]

[0018] where ||·|| is the standard squared l2 norm and s=N(f(t(x))) is the spatially varying adaptive uncertainty parameterized by a small linear network N; this transforms the mean squared error (MSE) loss into an appropriate likelihood function to handle uncertainty; this flexibility enables the network to learn to behave in situations where some outlier features cannot be upsampled.

[0019] In a preferred embodiment, in step S3, a flexible attention-based downsampler is selected; this component uses 1×1 convolution to predict a saliency map from high-resolution features, combines it with the learned spatially invariant weight kernel and bias kernel, and then normalizes the result to create a spatially varying blur kernel for feature interpolation; the specific calculation is as follows:

[0020] σ ↓ (F hr ) ij =softmax(w⊙Conv(F hr [Ω ij ])+b)·F hr [Ω ij ]

[0021] where σ ↓ (F hr ) ij is the i,jth component of the resulting feature map, F hr [Ω ij ] refers to the high-resolution feature block corresponding to the i, j position in the downsampled feature; w and b are the learning weights and bias kernels shared by all blocks, and ⊙ and · represent element-by-element multiplication and inner product, respectively.

[0022] In a preferred embodiment, in step S3, an implicit function F similar to NeRF is selected.hr =MLP(z) to parameterize the high-resolution features of a single image; the image coordinates and intensity are mapped to high-dimensional features at a given position through a multi-layer perceptron (MLP); in order to improve the spatial resolution of the implicit expression, in addition to the standard Fourier position features, Fourier color features are added, so that the network can utilize high-frequency color information from the original image and accelerate convergence; the specific calculation is as follows:

[0023]

[0024] in, represents the component discrete Fourier transform of the input signal z, is the frequency vector, e i and e j Represents a two-dimensional pixel coordinate field in the range of [-1, 1]; MLP is a three-layer ReLU network with dropout and layer normalization; In order to reduce memory and further accelerate the training of the FeatUp implicit network, this paper compresses the spatial variation features to the first 128 principal components through principal component analysis (PCA); This operation is almost lossless because the first 128 principal components explain about 96% of the variance in the DINO features of a single image.

[0025] In a preferred embodiment, in step S4, during the inference or optimization process, the DMTet method is used to extract isosurfaces from the neural field to approximate the three-dimensional boundary of the pig; the core advantage of the DMTet method is that it can not only generate triangular meshes from voxel grids or neural fields, but also directly optimize the parameters of the neural field by gradient descent, thereby adjusting the shape of the generated triangular mesh; at the same time, DMTet is suitable for processing objects that may undergo topological changes during deformation (such as changes in animal posture); specifically, the shape of the pig S = {x∈R 3 |s(x)=0} is defined as the signed distance function (SDF) s:R 3 →R isosurface; this function is implemented by a multi-layer perceptron (MLP), with the input as a 3D coordinate, which is initialized as the vertex coordinates of a tetrahedral mesh with a resolution of 128; to ensure that s is a signed distance function, randomly sample points in space And minimize the Equinox regularization term:

[0026]

[0027] Where X is a set of randomly sampled points, is the gradient growth at point x; for a valid signed distance function, its gradient The following conditions should be met:

[0028]

[0029] In a preferred embodiment, in step S5, DINO-ViT is used to extract feature descriptors. DINO-ViT is applied to the input image I∈R 3×H×W , and extract local key tags from the last attention layer and output tags In order to extract global features, a convolutional layer is further used to extract the global key mark F from it. k =f k (Φ k )∈R D and output marker F o =f o (Φ o )∈R D ;

[0030] The appearance of the image is learned using the Lambertian illumination model; the reflectance of the object is modeled in the same "canonical space" as the shape SDF and represented by a multi-layer perceptron (MLP); the input of the network is the global output feature F o and 3D vertex coordinates; the main light direction and ambient light and diffuse reflection intensity are predicted by a lighting network composed of a multi-layer perceptron (MLP); then the differentiable delayed grid rendering technology is adopted. The renderer first rasterizes all pixels overlapping with the 3D grid, calculates the 3D coordinates of the corresponding points in the canonical space, and then calculates the corresponding color through the lighting network; the specific loss calculation is as follows:

[0031]

[0032] Among them, dt(·) represents the distance transform, which is used to obtain a better gradient. is the render mask The intersection of the real mask M and λ dt is the weighting factor, I and are the true appearance of the image and the appearance predicted by the model, respectively.

[0033] In a preferred embodiment, in step S5, the pig shape information is learned. ins Shape and template V pr Slightly different, this difference is expressed by the deformation formula V ins,i =V pr,i +ΔV i Denotes that the displacement ΔV i Through a multi-layer perceptron (MLP) f ΔV (V pr,i ,F k ) According to the global image feature Fk Prediction; exploit the bilateral symmetry of the object by mirroring the query position on the yz plane to ensure that V ins and V pr Keep symmetry; at the same time, introduce the regularization term as follows:

[0034]

[0035] In a preferred embodiment, in step S5, the pig posture information is learned. In order to match the three-dimensional shape in the image and consider the deformation caused by the movement of the object skeleton joints, the instance mesh V ins Through the posture prediction function V i =g(V ins,i ,ξ) undergoes further deformation; different from ΔV i , where larger controllable deformations are allowed for easier processing; first, a simple heuristic method is used to initialize the joint positions of the standard pose on the instance mesh; except for the root joint, each joint has a parent joint, forming a tree structure; for the spine part, a skeleton consisting of 8 bones of equal length is estimated, which are located on two line segments, extending from the center of the standard pose to the two farthest vertices along the x-axis; the root joint is slightly raised, and four additional sets of bones are added to model the limbs; by identifying the lowest points of the four xz quadrants as the foot joints and finding the closest spine joints near each foot joint, they are connected to form a limb skeleton, and each leg consists of three bones of the same length; in order to connect the 3D mesh surface with the skeleton joints, a hybrid skin model is used, which is parameterized as B-1 joint rotations ξ b ∈SO(3) and view ξ1∈SE(3), where b=2,...,B,ξ1 is predicted by the view learning part; the mesh surface vertices are weighted by w ib When connected to a bone, these weights are defined based on the relative distance between the vertex and the bone; the vertex position is processed by the following linear blend skinning equation:

[0036]

[0037] Among them, ξ * represents the bone rotation in the standard posture; weight w i,b The definition is as follows:

[0038]

[0039] where d i,b Vertex V ins,i The minimum distance to each bone b (in world coordinates from the static pose joint positions and Definition), τ s is the temperature parameter set to 0.5;

[0040] Similar to instance-specific deformations, bone rotations are passed through the network f b According to the local feature Φ k and the global feature F k The prediction is performed for the pose joints, while the viewpoint is predicted separately; the action of the pose joints depends on the prediction of the local body joints, so the prediction effect can be improved by using local features; for this purpose, the center of gravity of each bone in the static pose is projected to the image plane, and local features are extracted from the key feature map of these projected pixel positions, which are defined as follows:

[0041]

[0042] Considering the interdependence between bones, a small transformer network is used to predict bone rotation, with B-1 labels v as input b =(F k ,Φ k (u b ),b,J b ,u b ), and output the bone rotation (ξ2,...,ξ B )=f b (v2,...,v B ), these rotations are expressed in Euler angles; learning the complex joint 3D pose of pigs without any explicit 3D geometric supervision through reconstruction loss from single view images is an ill-posed task; so in order to prevent unnatural poses, constraints are imposed on bone rotations: (1) all bone rotations are restricted to (-60°, 60°); (2) the rotations of the legs around the y-axis and z-axis are further restricted to (-6°, 6°) and (-18°, 18°), and the following regularizer is used to limit the degree of deviation from the rotation offset:

[0043]

[0044] In a preferred embodiment, in step S6, DINO features are rendered to supervise object viewpoint learning. The corresponding information is combined in an implicit way; specifically, by adding a coordinate network The self-supervised Vision Transformer features are fused into the 3D model; the network accepts a point x in the canonical space as input and outputs a D'-dimensional feature vector; this feature vector is shared among all instances of the same object category and therefore does not depend on the input image; these features are projected to the feature image via a differentiable renderer , and trained to minimize the rendering loss:

[0045]

[0046] In order to improve efficiency, Φ' k = PCA(Φ k ) is a version of the original DINO-ViT feature with principal component analysis (PCA) dimensionality reduction (D' = 16);

[0047] A major challenge in learning object perspectives is that there are multiple local optimal solutions in the reconstruction process. Although some studies have addressed this problem by sampling a large number of hypotheses, this requires rendering 3D models from all perspectives, which is computationally expensive and slow. Instead, a statistical method is used to explore four perspectives, but only one is sampled per iteration, which significantly reduces the computational cost. The perspective learning network of k Four view rotation hypotheses R are predicted in k ∈SO(3),k∈{1,2,3,4}, these hypotheses are located in one of the four quadrants around the object, and a score σ is predicted for each hypothesis k ; Calculate the probability p through softmax k ∝exp(-σ k / τ), select the best view k, where τ is a temperature parameter that decreases gradually during training; in each iteration, sample a hypothesis and minimize the target training score σ k To predict the expected reconstruction loss

[0048]

[0049] where sg[·] is the stop gradient operator, is the overall reconstruction loss under the sampled k-th hypothesis;

[0050] In a preferred embodiment, in step S7, the model training is performed by combining reconstruction loss, view loss and regularization term. Given a sample, including an image, a corresponding object mask and a corresponding DINO feature after super resolution, the training is performed in two steps: first, sampling the view hypothesis k and calculating the reconstruction loss:

[0051]

[0052] Then, the reconstruction loss, view loss, and regularization term are added together to obtain the total loss:

[0053]

[0054] Among them, λ im ,λ m ,λ f ,λ k ,λ E ,λ d,λ a ,λ h To balance the weights; calculate the gradient of L with respect to the model parameters and update these parameters; to improve sampling efficiency, only one perspective is selected for each gradient evaluation, and perspective sampling is performed based on the learned probability distribution.

[0055] In a preferred embodiment, in step S8, a cotangent Laplacian matrix is ​​used to achieve smoothing of the mesh, and a sparse Laplacian matrix is ​​constructed by calculating the connection relationship between each vertex in the mesh and the adjacent vertices; first, the vertex and face information of the mesh are extracted, and then the cosine value is used to quantify the relationship between adjacent vertices to fill the elements of the Laplacian matrix; at the same time, the area of ​​each face and its inverse value are calculated to normalize the influence of the vertex in the loss calculation; in this way, the geometric mutation of the mesh can be effectively reduced, and a smoother three-dimensional shape can be achieved, thereby improving the effect and visual quality of subsequent processing.

[0056] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0057] 1. In the present invention, the three-dimensional posture, shape and surface texture information of the pigs can be obtained, and the behavioral characteristics of the pigs can be more comprehensively reflected. The present invention adopts an implicit-explicit combined three-dimensional representation method, combined with high-dimensional data features optimized by super-resolution, and can reconstruct a high-density three-dimensional grid of pigs only through a single image. This is of great significance for practical application scenarios such as pig farms, which can save manpower and material resources and improve data acquisition efficiency. The present application can handle individual differences, posture changes and behavioral diversity of pigs. By training images of different pigs, different postures and behaviors, the model can learn the general shape and posture characteristics of pigs, and then predict the three-dimensional posture and shape of specific pigs. The present application optimizes the three-dimensional grid surface through the Laplace smoothing method to improve the reconstruction quality. This enables the method to adapt to the complex and changeable environment and pig behavior in the pig farm, and provides a more comprehensive and accurate data basis for pig behavior analysis.

[0058] 2. In the present invention, through three-dimensional reconstruction and posture analysis, abnormal behaviors of pigs, such as lameness, fighting, illness, etc., can be discovered in time, so that timely measures can be taken to intervene to avoid worsening of the disease or economic losses. Through three-dimensional reconstruction and body size analysis, the growth status of pigs, such as weight, body shape, etc., can be evaluated, so as to formulate a more scientific feeding plan and improve breeding efficiency and economic benefits. Through three-dimensional reconstruction and behavioral analysis, the behavioral characteristics and living habits of pigs can be better understood, so as to provide pigs with a more comfortable living environment and improve the welfare level of pigs. In short, this single-view deep learning-based pig Figure 3The reconstruction method provides important technical support for pig behavior analysis, can improve the efficiency and accuracy of pig behavior analysis, promote animal welfare and health management, and has important application value.

[0059] Description of the accompanying tables and graphs

[0060] Figure 1 It is a schematic diagram of the structure of the present invention;

[0061] Figure 2 It is the overall training framework of FeatUp in the present invention;

[0062] Figure 3 It is the up-sampling architecture in the present invention;

[0063] Figure 4 This is a visual comparison diagram of the original DINO feature, the DINO feature after bilinear interpolation, and the DINO feature + FeatUp in the present invention;

[0064] Figure 5 Comparison between the reconstructed 3D model and the rendering result in the present invention;

[0065] Figure 6 Different views of the reconstructed three-dimensional model of the present invention;

[0066] Figure 7 This is a comparison chart of the model evaluation results in the present invention; DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0068] Reference Figure 1-3 ,

[0069] Example:

[0070] A deep learning-based single-view Figure 3 The reconstruction method is divided into three parts.

[0071] S1: The first part involves the arrangement of data collection locations and cameras. The experimental data of the present invention were collected in a farm in Meizhou City, Guangdong Province, China in May 2023 and January 2024. The data mainly collected 6 three-way hybrid pigs, aged 30 to 150 days. The Hikvision camera was set up in the corner of the pigsty and shot towards the position of the pigs to collect RGB image data of different distances and light intensities, and then the data was preprocessed. Based on the data acquisition platform, a total of 7,621 RGB images of individual pigs standing or walking were obtained and selected for model training and 635 for model testing.

[0072] The second part involves feature extraction, using DINO-ViT to extract feature descriptors. DINO-ViT is applied to the input image and local key tags and output tags are extracted from the last attention layer; then the global key tags and output tags are further extracted using convolutional layers. Key tags are used to extract the correspondence of objects, such as the spatial position and relative shape of the target in the image, and the output tags are used to extract the appearance features of the object, such as color, texture, etc.

[0073] The third part first loads a tetrahedral mesh with a resolution of 128, and initializes an ellipsoidal signed distance function (SDF) and another signed distance function calculated by a multi-layer perceptron (MLP), and adds the two to obtain the total signed distance function. Next, DMTet is used to convert the implicit signed distance function into an explicit standard 3D mesh. Subsequently, the RGB image data is input into the network to update the standard 3D mesh with the learned posture and shape parameters to generate the final 3D mesh. The overall method flow chart is as follows Figure 1 As shown:

[0074] After obtaining RGB images of different directions, angles, and light intensities, labelimg was used to detect and roughly crop the pigs to obtain the approximate position of the pigs in the original image. The pigs were then annotated with masks using the eiseg annotation software. To improve efficiency and be suitable for real-time applications, the lightweight model static_hrnet18s_ocr48_cocolvis was selected. After completing the mask annotation, the image size was adjusted to 256×256 pixels. Finally, based on the RGB image and mask, a super-resolution network was used to extract and optimize the DINO features.

[0075] S2: DINO Feature Optimization for Super-Resolution: It is challenging to accurately determine the viewpoint of objects in training images before constructing a 3D surface, and the accuracy of the viewpoint is crucial for generating realistic 3D representations. Traditional methods usually rely on additional cues to assist viewpoint estimation, such as 2D keypoint correspondences. However, this experiment does not require these additional keypoint supervisions, but instead implicitly infers noisy correspondences by fusing the knowledge of DINO-ViT. Although this method can obtain good semantic information, it usually lacks sufficient spatial resolution and cannot capture fine spatial details. To address this problem, this paper introduces the FeatUp framework, which aims to recover the lost semantic information in deep features. Inspired by 3D reconstruction frameworks such as NeRF, FeatUp constructs an upsampler by observing multiple low-resolution feature views and applying a consistency loss on these feature maps. Method Training framework such as Figure 2 As shown, the specific architecture of the upsampler is as follows Figure 3 As shown:

[0076] The first step of the method is to perform small jitters on the input image, including padding, scaling, and horizontal flipping, and apply the DINO model to each transformed image to extract a series of low-resolution feature maps. These tiny image jitters enable subtle differences in the output features to be observed and provide sub-feature information for training the upsampler. Next, a consistent high-resolution feature map is constructed from these views, which is able to reproduce the "jitter" of the low-resolution features after downsampling. The downsampling method is similar to the rendering step in NeRF, but the difference is that it keeps track of the parameters of each image used for "jittering" and applies these transformations to the learned high-resolution features before downsampling. The downsampled features are then compared to the true model output using a Gaussian likelihood loss.

[0077] More specifically, let t∈T be a set of small transformations, such as margin, scaling, cropping, horizontal flipping, and their combinations. Let x be the input image, f be the model backbone, σ ↓ is the learned downsampler, σ ↑ is the learned upsampler. It can be evaluated by hr =σ ↑ (f(x),x) to form the high-resolution features for prediction. This parameterization allows σ ↑ as a guided upsampler (depends on both x and f(x)), an unguided upsampler (depends only on f(x)), an implicit network (depends only on x), or a learned feature buffer that does not depend on anything. The main multi-view reconstruction loss term can be formed as follows:

[0078]

[0079] where ||·|| is the standard squared l2 norm and s=N(f(t(x))) is the spatially varying adaptive uncertainty parameterized by a small linear network N (Hamilton et al., 2020). This transforms the mean squared error (MSE) loss into an appropriate likelihood function to handle uncertainty. This flexibility enables the network to learn to behave in situations where some anomalous features cannot be upsampled.

[0080] This paper chooses a flexible attention-based downsampler. This component uses 1×1 convolution to predict the saliency map from the high-resolution features and combines it with the learned spatially invariant weight kernel and bias kernel. The result is then normalized to create a spatially varying blur kernel for feature interpolation. The specific calculation is as follows:

[0081] σ ↓ (F hr ) ij=softmax(w⊙Conv(F hr [Ω ij ])+b)·F hr [Ω ij ]

[0082] where σ ↓ (F hr ) ij is the i,jth component of the resulting feature map, F hr [Ω ij ] refers to the high-resolution feature block corresponding to position i, j in the downsampled feature. w and b are the learning weights and bias kernels shared by all blocks, and ⊙ and · represent element-by-element multiplication and inner product, respectively.

[0083] In addition, this paper chooses an implicit function F similar to NeRF hr =MLP(z) to parameterize the high-resolution features of a single image. The image coordinates and intensity are mapped to high-dimensional features at a given position through a multi-layer perceptron (MLP). In order to improve the spatial resolution of the implicit expression, in addition to the standard Fourier position features, Fourier color features are added, allowing the network to utilize high-frequency color information from the original image and accelerate convergence. The specific calculation is as follows:

[0084]

[0085] in, represents the component discrete Fourier transform of the input signal z, is the frequency vector, e i and e j Represents a two-dimensional pixel coordinate field in the range of [-1, 1]. MLP is a three-layer ReLU network with dropout and layer normalization. In order to reduce memory and further accelerate the training of the FeatUp implicit network, this paper compresses the spatial variation features to the first 128 principal components through principal component analysis (PCA). This operation is almost lossless because the first 128 principal components explain about 96% of the variance in the DINO features of a single image.

[0086] S3: Pig 3D pose and surface reconstruction

[0087] The goal of this experiment is to train a monocular 3D reconstruction network given a collection of single-view images of pigs collected from a pig farm. The network is able to predict the joint pose, 3D shape, and surface texture of a pig from a single image. During training, all that is required is a 2D foreground segmentation mask of the pig and a simple description of the pig's skeleton topology, such as the number of legs.

[0088] The choice of a representation for the 3D shape of an object is crucial and must satisfy the following two conditions: (1) it must be sufficiently regular to support weakly supervised learning via image reconstruction and (2) it must be sufficiently expressive to model fine shape details and deformations. Existing 3D representation methods can be divided into two categories: explicit and implicit. Explicit representations (e.g., triangle meshes) are more compact, easily support powerful deformation models (e.g., hybrid skinning), and have easy-to-control geometric smoothness. However, these methods are often difficult to optimize during training and are prone to defects such as wrinkles, usually requiring remeshing. Implicit representations (e.g., volumetric, signed distance fields, and radiance fields) usually require a lot of supervision but are amenable to gradient-based optimization and can handle topological changes. In this experiment, a representation method that combines the advantages of both is adopted, which enables the method to learn complex 3D shapes and deformations through only weakly supervised image reconstruction without the need for dense viewpoint coverage or 3D ground truth data. Specifically, the shape of the pig is implicitly represented by a neural field and converted to an explicit mesh in real time using the differentiable marching tetrahedron (DMTet) method. The neural field is learned by a neural network and can capture both global and local features of the shape. During the inference or optimization process, the DMTet method is used to extract isosurfaces from the neural field to approximate the three-dimensional boundary of the pig. The core advantage of the DMTet method is that it can not only generate triangular meshes from voxel grids or neural fields, but also directly optimize the parameters of the neural field by gradient descent to adjust the shape of the generated triangular mesh. At the same time, DMTet is suitable for processing objects that may undergo topological changes during deformation (such as changes in animal posture). Specifically, the shape of the pig S = {x∈R 3 |s(x)=0} is defined as the signed distance function (SDF) s:R 3 →R isosurface. This function is implemented by a multi-layer perceptron (MLP) with input as 3D coordinates initialized to vertex coordinates of a tetrahedral mesh with a resolution of 128. To ensure that s is a signed distance function, this system randomly samples points in space. And minimize the Equinox regularization term:

[0089]

[0090] Where X is a set of randomly sampled points, is the gradient growth at point x. For a valid signed distance function, its gradient The following conditions should be met:

[0091]

[0092] The experiment then evaluates the function s at the vertices of the tetrahedral mesh and extracts a watertight 3D mesh from the signed distance function using the fast and differentiable DMTet. The function s is initialized with a general ellipsoid stretched along the z-axis. The tetrahedral mesh used to initialize the 3D coordinates and the triangular mesh extracted on the surface of the ellipsoid signed distance function are as follows: Figure 4 and Figure 5 As shown:

[0093] This experiment first uses implicit-explicit combined representation to learn a category template V shared by all instances pr , and predict the 3D instance model V from a single image ins To match the shape and posture in the input image. The input image is encoded by the self-supervised Vision Transformer (DINO-ViT), which outputs a set of key tags and output tags. The key tags are used to extract the correspondence between objects, such as the spatial position and relative shape of the target in the image, and the output tags are used to extract the appearance features of the object, such as color, texture, etc. Specifically, this system applies DINO-ViT to the input image I∈R 3×H×W , and extract local key tags from the last attention layer and output tags In order to extract global features, this system further uses a convolutional layer to extract global key tags F k =f k (Φ k )∈R D and output marker F o =f o (Φ o )∈R D .

[0094] To learn the appearance of an image, this system uses the Lambertian illumination model. The reflectance of the object is modeled in the same "canonical space" as the shape SDF and represented by a multi-layer perceptron (MLP). The input of the network is the global output feature F o and 3D vertex coordinates. The main light direction and ambient light and diffuse reflection intensity are predicted by a lighting network composed of a multi-layer perceptron (MLP). Then, using the differentiable delayed grid rendering technology, the renderer first rasterizes all pixels overlapping with the 3D grid, calculates the 3D coordinates of the corresponding points in the canonical space, and then calculates the corresponding color through the lighting network. The specific loss calculation is as follows:

[0095]

[0096] Among them, dt(·) represents the distance transform, which is used to obtain a better gradient. is the render mask The intersection of the real mask M and λ dt is the weighting factor, I and are the true appearance of the image and the appearance predicted by the model, respectively.

[0097] In the early stages of network training, our system first learns the viewpoint prediction of the object, which is a key step in learning 3D shapes, but is usually difficult. To this end, our system uses self-supervised correspondences to perform multi-hypothesis viewpoint prediction. Since learning explicit correspondences is difficult and fragile under unsupervised conditions, our system incorporates correspondence information in an implicit way. Specifically, our system adds a coordinate network The self-supervised Vision Transformer features are fused into the 3D model. The network accepts as input a point x in the canonical space and outputs a D'-dimensional feature vector. This feature vector is shared between all instances of the same object category and therefore does not depend on the input image. These features are projected to the feature image via a differentiable renderer , and trained to minimize the rendering loss:

[0098]

[0099] In order to improve efficiency, Φ' k = PCA(Φ k ) is a version of the original DINO-ViT feature with principal component analysis (PCA) dimensionality reduction (D'=16).

[0100] A major challenge in learning object perspectives is that there are multiple local optima in the reconstruction process. While some studies have addressed this problem by sampling a large number of hypotheses, this requires rendering 3D models from all perspectives, which is computationally expensive and slow. In contrast, our system uses a statistical approach to explore four perspectives, but only samples one per iteration, significantly reducing computational cost. Our perspective learning network extracts the optimal local optima from the global feature F. k Four view rotation hypotheses R are predicted in k ∈SO(3),k∈{1,2,3,4}, these hypotheses are located in one of the four quadrants around the object, and a score σ is predicted for each hypothesis k . Calculate the probability p through softmax k ∝exp(-σ k / τ), select the best view k, where τ is a temperature parameter that decreases during training. In each iteration, the system samples a hypothesis and minimizes the target training score σ k To predict the expected reconstruction loss

[0101]

[0102] where sg[·] is the stop gradient operator, is the overall reconstruction loss under the k-th hypothesis sampled.

[0103] Instance mesh V for each object ins Shape and template V pr Slightly different, this difference is expressed by the deformation formula V ins,i =V pr,i +ΔV i Denotes that the displacement ΔV i Through a multi-layer perceptron (MLP) f ΔV (V pr,i ,F k ) According to the global image feature F k Prediction. This system exploits the bilateral symmetry of the object by mirroring the query position on the yz plane to ensure that V ins and V pr Keep symmetry. At the same time, the regularization term is introduced as follows:

[0104]

[0105] S4: To match the 3D shape in the image and take into account the deformation caused by the motion of the object skeleton joints, the instance mesh V ins Through the posture prediction function V i =g(V ins,i ,ξ) undergoes further deformation. Different from ΔV i , where larger controllable deformations are allowed for easier processing. The system first initializes the joint positions of the standard pose on the instance mesh using a simple heuristic method. Except for the root joint, each joint has a parent joint, forming a tree-like structure. For the spine, the system estimates a skeleton consisting of 8 bones of equal length, which are located on two line segments extending from the center of the standard pose to the two farthest vertices along the x-axis. The root joint is slightly raised, and four additional sets of bones are added to model the limbs. By identifying the lowest points of the four xz quadrants as the foot joints and finding the closest spine joints near each foot joint, they are connected to form a limb skeleton, with each leg consisting of three bones of the same length. In order to connect the 3D mesh surface to the skeleton joints, the system uses a hybrid skin model, which is parameterized as B-1 joint rotations ξ b ∈SO(3) and view ξ1∈SE(3), where b=2,...,B,ξ1 is predicted by the view learning part. The mesh surface vertices are weighted by w ib When connected to a bone, these weights are defined based on the relative distance between the vertex and the bone. The vertex position is processed by the following linear blend skinning equation:

[0106]

[0107] Among them, ξ * Represents the bone rotation in the standard posture. Weight w i,b The definition is as follows:

[0108]

[0109] where d i,b Vertex V ins,i The minimum distance to each bone b (in world coordinates from the static pose joint positions and Definition), τ s is the temperature parameter set to 0.5.

[0110] Similar to instance-specific deformations, bone rotations are passed through the network f b According to the local feature Φ k and the global feature F k The prediction is done for the pose joints, while the viewpoint is predicted separately. The actions of the pose joints depend on the prediction of the local body joints, so using local features can improve the prediction. To this end, the center of gravity of each bone in the static pose is projected to the image plane, and local features are extracted from the key feature map of these projected pixel positions, which are defined as follows:

[0111]

[0112] Considering the interdependence between bones, this system uses a small transformer network to predict bone rotation, with B-1 labels v as input b =(F k ,Φ k (u b ),b,J b ,u b ), and output the bone rotation (ξ2,...,ξ B )=f b (v2,...,v B ), these rotations are expressed in Euler angles. Learning the complex joint 3D pose of pigs through reconstruction loss from single-view images without any explicit 3D geometric supervision is an ill-posed task. So to prevent unnatural poses, this system imposes constraints on bone rotations: (1) All bone rotations are restricted to (-60°, 60°); (2) The rotations of the legs around the y-axis and z-axis are further restricted to (-6°, 6°) and (-18°, 18°). At the same time, the following regularizer is used to limit the degree of deviation from the rotation offset:

[0113]

[0114] Given a sample, including an image, the corresponding object mask, and the corresponding super-resolved DINO features, the training is performed in two steps. First, sample the view hypothesis k and calculate the reconstruction loss:

[0115]

[0116] Then, the reconstruction loss, view loss, and regularization term are added together to obtain the total loss:

[0117]

[0118] Among them, λ im ,λ m ,λ f ,λ k ,λ E ,λ d ,λ a ,λ h To balance the weights. Calculate the gradient of L with respect to the model parameters and update these parameters. To improve sampling efficiency, only one perspective is selected for each gradient evaluation, and perspective sampling is performed based on the learned probability distribution.

[0119] S5: 3D surface smoothing of pigs: In this study, the system uses the cotangent Laplacian matrix to achieve mesh smoothing. This method constructs a sparse Laplacian matrix by calculating the connection relationship between each vertex and adjacent vertices in the mesh. Specifically, the system first extracts the vertex and face information of the mesh, and then uses the cosine value to quantify the relationship between adjacent vertices to fill the elements of the Laplacian matrix. At the same time, the system calculates the area of ​​each face and its inverse value in order to normalize the influence of the vertex in the loss calculation. In this way, the system can effectively reduce the geometric mutation of the mesh and achieve a smoother 3D shape, thereby improving the effect and visual quality of subsequent processing.

[0120] The test platform used in the present invention is Ubuntu 20.04 operating system, the CPU model is Intel Core i5-12600kf, the memory capacity is 16GB, the graphics card GPU model is Nvidia GeForce RTX3080, and the video memory capacity is 10GB. In the Pytorch deep learning framework, the Python programming language is used to implement the present invention. During the model training process, Batchsize is set to 1, the initial learning rate is set to 0.001, the total iteration round is set to 120, the skeleton learning start iteration round is set to 3, the shape learning start iteration round is set to 50, and the texture learning start iteration round is set to 1.

[0121] The present invention uses TransferPCK as an evaluation index for the model reconstruction effect. PCK is an index used to evaluate key point detection tasks. It calculates the ratio of the distance between the detected key points and the true key points that is less than a certain threshold. TransferPCK calculates PCK on different domains or different data sets to evaluate the generalization ability of the model. The accuracy of the method proposed in the present invention on the PCK@0.1 evaluation index is 74.86%, which is 1.75% higher than the original model, improving the generalization ability and reconstruction quality of the model.

[0122] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0123] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A single-view 3D reconstruction method for pigs based on deep learning, characterized by: The method comprises the following steps: S1: Arrange the cameras first and collect data; S2: The data is then subjected to object detection and segmentation, and the image and mask are resized to a resolution of 256×256. S3: Use the FeatUp framework to improve the spatial resolution of DINO features and capture subtle spatial details; S4: Use signed distance function (SDF) to implicitly represent the pig shape, and use differentiable marching tetrahedron (DMTet) to convert SDF into explicit mesh; S5: Use DINO-ViT to extract feature descriptors of RGB images and learn the pig skeleton structure, shape and appearance information; S6: Generate pig surface texture map, image mask and DINO features using Lambertian illumination model and differentiable deferred mesh rendering technology; S7: Combine reconstruction loss, view loss and regularization term to train the model; S8: Use the cosine Laplacian matrix to smooth the mesh and reduce geometric abruptness.

2. A method for three-dimensional reconstruction of a pig from a single view based on deep learning as claimed in claim 1, characterized in that: In step S1, the camera is directed toward the position of the pig to capture RGB image data of different directions, angles and light intensities.

3. The method for three-dimensional reconstruction of a pig based on single view based on deep learning as claimed in claim 1, characterized in that: In step S2, labelimg is used to label and crop the image, eiseg is used to label the cropped image with a mask, and then the size of the cropped image and the mask is adjusted to 256×256 according to the image mask.

4. A method for three-dimensional reconstruction of a pig from a single view based on deep learning as claimed in claim 1, characterized in that: In step S3, the FeatUp framework is used to construct an upsampler by observing multiple low-resolution feature views and applying consistency loss to these feature maps, the input image is slightly jittered, including padding, scaling, and horizontal flipping operations, and the DINO model is applied to each transformed image to extract a series of low-resolution feature maps; these slight image jitters enable subtle differences in the output features to be observed and provide sub-feature information for training the upsampler; Next, a consistent high-resolution feature map is constructed from these views, which is able to reproduce the "jittering" of the low-resolution features after downsampling; the downsampling method is similar to the rendering step in NeRF, but the difference is that it keeps track of the parameters of each image used for "jittering" and applies these transformations to the learned high-resolution features before downsampling; the downsampled features are then compared to the true model output using a Gaussian likelihood loss; More specifically, let t∈T be a set of small transformations, such as margin, scaling, cropping, horizontal flipping, and their combinations; let x be the input image, f be the model backbone, σ ↓ is the learned downsampler, σ ↑ is the learned upsampler; it can be evaluated by hr =σ ↑ (f(x),x) to form the high-resolution features for prediction; this parameterization allows σ ↑ As a guided upsampler (depends on both x and f(x)), an unguided upsampler (depends only on f(x)), an implicit network (depends only on x), or a learned feature buffer that does not depend on anything; the main multi-view reconstruction loss term can be formed as follows: where ||·|| is the standard squared l2 norm and s=N(f(t(x))) is the spatially varying adaptive uncertainty parameterized by a small linear network N; this transforms the mean squared error (MSE) loss into an appropriate likelihood function to handle uncertainty; this flexibility enables the network to learn to behave in situations where some outlier features cannot be upsampled.

5. The method for three-dimensional reconstruction of a pig from a single view based on deep learning as claimed in claim 1, characterized in that: In step S3, a flexible attention-based downsampler is selected; this component uses 1×1 convolution to predict the saliency map from the high-resolution features, combines it with the learned spatially invariant weight kernel and bias kernel, and then normalizes the result to create a spatially varying blur kernel for feature interpolation; the specific calculation is as follows: s ↓ (F hr ) ij =softmax(w⊙Conv(F hr [Oh ij ])+b)·F hr [Oh ij ] where σ ↓ (F hr ) ij is the i,jth component of the resulting feature map, F hr [Ω ij ] refers to the high-resolution feature block corresponding to the i, j position in the downsampled feature; w and b are the learning weights and bias kernels shared by all blocks, and ⊙ and · represent element-by-element multiplication and inner product, respectively.

6. The method for three-dimensional reconstruction of a pig from a single view based on deep learning as claimed in claim 1, characterized in that: In step S3, an implicit function F similar to NeRF is selected. hr =MLP(z) to parameterize the high-resolution features of a single image; the image coordinates and intensity are mapped to high-dimensional features at a given position through a multi-layer perceptron (MLP); in order to improve the spatial resolution of the implicit expression, in addition to the standard Fourier position features, Fourier color features are added, so that the network can utilize high-frequency color information from the original image and accelerate convergence; the specific calculation is as follows: in, represents the component discrete Fourier transform of the input signal z, is the frequency vector, e i and e j Represents a two-dimensional pixel coordinate field in the range of [-1, 1]; MLP is a three-layer ReLU network with dropout and layer normalization; In order to reduce memory and further accelerate the training of the FeatUp implicit network, this paper compresses the spatial variation features to the first 128 principal components through principal component analysis (PCA); This operation is almost lossless because the first 128 principal components explain about 96% of the variance in the DINO features of a single image.

7. The method for three-dimensional reconstruction of a pig based on single view based on deep learning as claimed in claim 1, characterized in that: In step S4, during the inference or optimization process, the DMTet method is used to extract isosurfaces from the neural field to approximate the three-dimensional boundary of the pig; the core advantage of the DMTet method is that it can not only generate a triangular mesh from a voxel grid or a neural field, but also directly optimize the parameters of the neural field by gradient descent, thereby adjusting the shape of the generated triangular mesh; at the same time, DMTet is suitable for processing objects that may undergo topological changes during deformation (such as changes in animal posture); specifically, the shape of the pig S = {x∈R 3 |s(x)=0} is defined as the signed distance function (SDF) s:R 3 →R isosurface; this function is implemented by a multi-layer perceptron (MLP), with the input as a 3D coordinate, which is initialized as the vertex coordinates of a tetrahedral mesh with a resolution of 128; to ensure that s is a signed distance function, randomly sample points in space And minimize the Equinox regularization term: Where X is a set of randomly sampled points, is the gradient growth at point x; for a valid signed distance function, its gradient The following conditions should be met:

8. The method for three-dimensional reconstruction of a pig based on single view based on deep learning as claimed in claim 1, characterized in that: In step S5, DINO-ViT is used to extract feature descriptors and DINO-ViT is applied to the input image I∈R 3×H×W , and extract local key tags from the last attention layer and output tags In order to extract global features, a convolutional layer is further used to extract the global key mark F from it. k =f k (Φ k )∈R D and output marker F o =f o (Φ o )∈R D ; The appearance of the image is learned using the Lambertian illumination model; the reflectance of the object is modeled in the same "canonical space" as the shape SDF, represented by a multi-layer perceptron (MLP); the input of the network is the global output feature F o and 3D vertex coordinates; the main light direction and ambient light and diffuse reflection intensity are predicted by a lighting network composed of a multi-layer perceptron (MLP); then the differentiable delayed grid rendering technology is adopted. The renderer first rasterizes all pixels overlapping with the 3D grid, calculates the 3D coordinates of the corresponding points in the canonical space, and then calculates the corresponding color through the lighting network; the specific loss calculation is as follows: Among them, dt(·) represents the distance transform, which is used to obtain a better gradient. is the render mask The intersection of the real mask M and λ dt is the weighting factor, I and are the true appearance of the image and the appearance predicted by the model, respectively.

9. The method for three-dimensional reconstruction of a pig from a single view based on deep learning as claimed in claim 1, characterized in that: In step S5, the shape information of the pigs is learned, and the instance mesh V of each object is ins Shape and template V pr Slightly different, this difference is expressed by the deformation formula V ins,i =V pr,i +ΔV i Denotes that the displacement ΔV i Through a multi-layer perceptron (MLP) f ΔV (V pr,i ,F k ) According to the global image feature F k Prediction; exploit the bilateral symmetry of the object by mirroring the query position on the yz plane to ensure that V ins and V pr Keep symmetry; at the same time, introduce the regularization term as follows:

10. The method for three-dimensional reconstruction of a pig based on single view based on deep learning as claimed in claim 1, characterized in that: In step S5, the pig posture information is learned. In order to match the three-dimensional shape in the image and consider the deformation caused by the movement of the object skeleton joints, the instance mesh V ins Through the posture prediction function V i =g(V ins,i ,ξ) undergoes further deformation; different from ΔV i , where larger controllable deformations are allowed for easier processing; first, a simple heuristic method is used to initialize the joint positions of the standard pose on the instance mesh; Except for the root joint, each joint has a parent joint to form a tree-like structure. For the spine, a skeleton consisting of 8 bones of equal length is estimated. These bones are located on two line segments, extending from the center of the standard posture to the two farthest vertices along the x-axis. The root joint is slightly raised, and four additional sets of bones are added to model the limbs. By identifying the lowest points of the four xz quadrants as the foot joints and finding the closest spine joints near each foot joint, they are connected to form a limb skeleton, and each leg consists of three bones of the same length. In order to connect the 3D mesh surface to the skeleton joints, a hybrid skin model is used, which is parameterized as B-1 joint rotations ξ. b ∈SO(3) and view ξ1∈SE(3), where b=2,...,B,ξ1 is predicted by the view learning part; the mesh surface vertices are weighted by w ib When connected to a bone, these weights are defined based on the relative distance between the vertex and the bone; the vertex position is processed by the following linear blend skinning equation: Among them, ξ * represents the bone rotation in the standard posture; weight w i,b The definition is as follows: where d i,b Vertex V ins,i The minimum distance to each bone b (in world coordinates from the static pose joint positions and Definition), τ s is the temperature parameter set to 0.5; Similar to instance-specific deformations, bone rotations are passed through the network f b According to the local feature Φ k and the global feature F k The prediction is performed for the pose joints, while the viewpoint is predicted separately; the action of the pose joints depends on the prediction of the local body joints, so the prediction effect can be improved by using local features; for this purpose, the center of gravity of each bone in the static pose is projected to the image plane, and local features are extracted from the key feature map of these projected pixel positions, which are defined as follows: Considering the interdependence between bones, a small transformer network is used to predict bone rotation, with B-1 labels v as input b =(F k ,Φ k (u b ),b,J b ,u b ), and output the bone rotation (ξ2,...,ξ B ) = f b (v2,...,v B ), these rotations are expressed in Euler angles; learning the complex joint 3D pose of pigs without any explicit 3D geometric supervision through reconstruction loss from single view images is an ill-posed task; so in order to prevent unnatural poses, constraints are imposed on bone rotations: (1) all bone rotations are restricted to (-60°, 60°); (2) the rotations of the legs around the y-axis and z-axis are further restricted to (-6°, 6°) and (-18°, 18°), and the following regularizer is used to limit the degree of deviation from the rotation offset: In step S6, DINO features are rendered to supervise object viewpoint learning, and the corresponding information is combined in an implicit way; specifically, by adding a coordinate network The self-supervised VisionTransformer features are fused into the 3D model; the network accepts a point x in the canonical space as input and outputs a D'-dimensional feature vector; this feature vector is shared among all instances of the same object category and therefore does not depend on the input image; these features are projected to the feature image via a differentiable renderer , and trained to minimize the rendering loss: In order to improve efficiency, Φ' k = PCA(Φ k ) is a version of the original DINO-ViT feature with principal component analysis (PCA) dimensionality reduction (D' = 16); A major challenge in learning object perspectives is that there are multiple local optimal solutions in the reconstruction process. Although some studies have addressed this problem by sampling a large number of hypotheses, this requires rendering 3D models from all perspectives, which is computationally expensive and slow. Instead, a statistical method is used to explore four perspectives, but only one is sampled per iteration, which significantly reduces the computational cost. The perspective learning network of k Four view rotation hypotheses R are predicted in k ∈SO(3),k∈{1,2,3,4}, these hypotheses are located in one of the four quadrants around the object, and a score σ is predicted for each hypothesis k ; Calculate the probability p through softmax k ∝exp(-σ k / τ), select the best view k, where τ is a temperature parameter that decreases gradually during training; in each iteration, sample a hypothesis and minimize the target training score σ k To predict the expected reconstruction loss where sg[·] is the stop gradient operator, is the overall reconstruction loss under the sampled k-th hypothesis; in step S7, the reconstruction loss, the view loss and the regularization term are combined to perform model training. Given a sample, including an image, a corresponding object mask and a corresponding DINO feature after super resolution, the training is performed in two steps; first, the view hypothesis k is sampled and the reconstruction loss is calculated: Then, the reconstruction loss, view loss, and regularization term are added together to obtain the total loss: Among them, λ im ,λ m ,λ f ,λ k ,λ E ,λ d ,λ a ,λ h To balance the weights; calculate the gradient of L with respect to the model parameters, and update these parameters; to improve sampling efficiency, only one perspective is selected for each gradient evaluation, and perspective sampling is performed based on the learned probability distribution. In step S8, a cosine Laplace matrix is ​​used to achieve mesh smoothing, and a sparse Laplace matrix is ​​constructed by calculating the connection relationship between each vertex and adjacent vertices in the mesh; first, the vertex and face information of the mesh are extracted, and then the cosine value is used to quantify the relationship between adjacent vertices, thereby filling the elements of the Laplace matrix; at the same time, the area of ​​each face and its inverse value are calculated to normalize the influence of the vertex in the loss calculation.

Images