Object physical attribute prediction method and device, equipment and storage medium

Abstract

The invention discloses an object physical attribute prediction method and device, equipment and a storage medium, and relates to the technical field of computer vision and artificial intelligence, and the method comprises the steps: obtaining a multi-view picture of a target object; inputting the multi-view picture into a vision-language model to obtain a physical attribute dictionary; reconstructing the 3DGS based on a multi-view picture in combination with a loss mechanism to obtain a three-dimensional Gaussian point cloud representation; embedding region perception features in the three-dimensional Gaussian point cloud representation, and carrying out joint supervision training to obtain semantic region perception features; and clustering and semantic reasoning are carried out on the Gaussian point clouds based on the semantic region perception features, a physical attribute dictionary is queried, and a physical attribute value corresponding to each Gaussian point cloud is obtained, so that the semantic accuracy and cross-material generalization ability of point cloud level physical attribute prediction can be remarkably improved while the geometric reconstruction quality is guaranteed, and the physical attribute prediction efficiency is improved. And zero-sample and fine-grained physical perception is supported.

Description

Technical Field

[0001] This application relates to the fields of computer vision and artificial intelligence, and in particular to methods, apparatus, devices, and storage media for predicting the physical properties of objects. Background Technology

[0002] Existing neural 3D reconstruction techniques for physical property perception mainly rely on NeRF (Neural Radiance Fields) and 3D Gaussian Splatting (3DGS) for 3D reconstruction, combined with pre-trained visual-language models (VLMs), to achieve zero-shot physical property prediction without explicit annotation, providing preliminary support for automatic estimation of object mass and material. However, current methods still have two key problems: Firstly, NeRF reconstructs objects based on NeRF and embeds clips into each point cloud. However, the clip feature map of the 2D image itself does not have the ability to segment boundaries. Although it can basically correctly infer the approximate position at the semantic level, its segmentation effect is poor. Secondly, for objects reconstructed based on 3DGS, the 3D reconstruction results are prone to geometric collapse, surface holes or floating artifacts, especially for objects with weak textures or simple geometric structures. PUGS, which reconstructs objects based on 3DGS, embeds region segmentation features into the point cloud and has good segmentation capabilities. However, due to the chaotic masking of the segmentation model and the lack of semantics, inference errors may occur.

[0003] Therefore, there is an urgent need for a physical property prediction scheme that can guarantee the quality of 3D reconstruction while possessing stronger semantic discrimination and broader material perception capabilities.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, device, and storage medium for predicting the physical properties of objects, aiming to solve the technical problem of how to improve the semantic accuracy and cross-material generalization ability of point cloud-level physical property prediction while ensuring the quality of three-dimensional geometric reconstruction.

[0006] To achieve the above objectives, this application proposes a method for predicting the physical properties of an object, the method comprising: Obtain multi-view images of the target object; The multi-view images are input into a pre-trained visual-language model to obtain a physical attribute dictionary; Based on the multi-view images and combined with a preset loss mechanism, the 3D Gaussian splash 3DGS is reconstructed to obtain a 3D Gaussian point cloud representation. Region-aware features are embedded in the 3D Gaussian point cloud representation, and joint supervised training is performed on the 3D Gaussian point cloud representation to obtain semantic region-aware features. Based on the semantic region perception features, clustering and semantic reasoning are performed on Gaussian point clouds, and the physical attribute dictionary is queried to obtain the physical attribute value corresponding to each Gaussian point cloud.

[0007] In one embodiment, the step of reconstructing a 3D Gaussian splash 3DGS based on the multi-view images and in conjunction with a preset loss mechanism to obtain a 3D Gaussian point cloud representation includes: Calculate the boundary-aware loss function between the rendered transparent image corresponding to the multi-view image and the mask of the real object; Calculate the local gradient of the rendering depth map corresponding to the multi-view image to obtain the smoothing loss function; Based on the normalized image gradient and corresponding depth information of each pixel in the multi-view image, the surface normal in the image space is estimated, and combined with the geometric normal calculated by the three-dimensional Gaussian point cloud, a normal vector is constructed. By combining the boundary-aware loss function, the smoothing loss function, and the normal vector, geometric constraints are applied to the reconstruction process of the 3D Gaussian splash, and the 3D Gaussian point cloud representation is constructed.

[0008] In one embodiment, the reconstruction process of the 3D Gaussian splash is geometrically constrained by combining a boundary-aware loss function, a smoothing loss function, and a normal vector. The steps for constructing the 3D Gaussian point cloud representation include: The boundary-aware loss, the smoothing loss function, and the normal vector are summed to obtain the total geometric reconstruction loss function; The reconstruction process of the 3D Gaussian splash is geometrically constrained by the total geometric reconstruction loss function to construct the 3D Gaussian point cloud representation.

[0009] In one embodiment, the step of embedding region-aware features into the 3D Gaussian point cloud representation and performing joint supervised training on the 3D Gaussian point cloud representation to obtain semantic region-aware features includes: The region-aware features are embedded into each 3D Gaussian point cloud, and the region-aware features are converted into a target dimension feature map through rasterization. The target dimension feature map is divided into a first sub-dimensional feature map and a second sub-dimensional feature map; For the first sub-dimensional feature map, the multi-scale segmentation mask extracted from the multi-view image by the pre-trained SAM model for segmenting everything is used as the first supervision target; For the second sub-dimensional feature map, the semantic feature map and structural feature map extracted from the multi-view image are used as the second supervision target; By combining the first supervisory target, the second supervisory target, the loss function corresponding to the first supervisory target, and the loss function of the second supervisory target, the semantic region-aware features are obtained by training the three-dimensional Gaussian point cloud.

[0010] In one embodiment, before the step of training the 3D Gaussian point cloud to obtain semantic region-aware features by jointly using the first supervisory target, the second supervisory target, the loss function corresponding to the first supervisory target, and the loss function of the second supervisory target, the following steps are included: Calculate the feature similarity between any two pixels in the first sub-dimensional feature map; Determine whether the pixel belongs to the same semantic region defined by the multi-scale segmentation mask; If so, a consistency loss term is constructed based on the feature similarity to constrain pixel features in the same semantic region to tend to be similar; If not, a discriminative loss term is constructed based on the feature similarity to constrain the differences in pixel feature regions of different semantic regions; The consistency loss term and the discriminative loss term are combined into a region-aware loss function, and the region-aware loss function is used as the loss function for the first supervised target.

[0011] In one embodiment, before the step of training the 3D Gaussian point cloud to obtain semantic region-aware features by jointly using the first supervised target, the second supervised target, the loss function corresponding to the first supervised target, and the loss function of the second supervised target, the method further includes: Calculate the cosine similarity between the CLIP semantic feature map and the DINOv2 structural feature map; A semantic alignment loss term is constructed based on the cosine similarity; The semantic alignment term is used as the loss function for the second supervision objective.

[0012] In one embodiment, the step of clustering and semantically reasoning the Gaussian point clouds based on the semantic region-aware features, and querying the physical attribute dictionary to obtain the physical attribute value corresponding to each Gaussian point cloud includes: The semantic region-aware features are fused with the Gaussian point cloud to obtain latent features; The DBSCAN algorithm is used to embed the location information of the Gaussian point cloud into the latent features for clustering to obtain cluster centers; The semantic region-aware features of the cluster centers are input into a pre-trained mapping model to generate corresponding CLIP semantic features and DINOv2 structural features. Calculate the cosine similarity between the CLIP semantic features and the CLIP features of the preset material category text description, and determine the material category corresponding to the cluster center based on the maximum similarity; Based on the material category, the corresponding physical property value is queried from the physical property dictionary.

[0013] Furthermore, to achieve the above objectives, this application also proposes an object physical property prediction device, the object physical property prediction device comprising: The image acquisition module is used to acquire multi-view images of the target object; The image processing module is used to input the multi-view images into a pre-trained visual-language model to obtain a physical attribute dictionary; The reconstruction module is used to reconstruct the three-dimensional Gaussian splash 3DGS based on the multi-view image and in combination with a preset loss mechanism to obtain a three-dimensional Gaussian point cloud representation. The training module is used to embed region-aware features into the 3D Gaussian point cloud representation and perform joint supervised training on the 3D Gaussian point cloud representation to obtain semantic region-aware features. The reasoning module is used to perform clustering and semantic reasoning on Gaussian point clouds based on the semantic region perception features, and to query the physical attribute dictionary to obtain the physical attribute value corresponding to each Gaussian point cloud.

[0014] In addition, to achieve the above objectives, this application also proposes an object physical property prediction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the object physical property prediction method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the object physical property prediction method as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the object physical property prediction method as described above.

[0017] This application proposes a method, apparatus, device, and storage medium for predicting the physical properties of an object. The method includes: acquiring multi-view images of a target object; inputting the multi-view images into a pre-trained visual-language model to obtain a physical property dictionary; reconstructing a 3DGS based on the multi-view images and combining a preset loss mechanism to obtain a 3D Gaussian point cloud representation; embedding region-aware features into the 3D Gaussian point cloud representation and performing joint supervised training on the 3D Gaussian point cloud representation to obtain semantic region-aware features; performing clustering and semantic reasoning on the Gaussian point clouds based on the semantic region-aware features, querying the physical property dictionary, and obtaining the physical property value corresponding to each Gaussian point cloud. This scheme, by embedding region-aware features into the 3D Gaussian point cloud representation and performing joint supervised training on the point cloud representation using multi-source supervision signals, obtains semantic region-aware features with component-level semantic discrimination capabilities, which can improve the semantic accuracy of point cloud-level physical property prediction while ensuring the quality of 3D geometric reconstruction; and by constructing a physical property dictionary based on a pre-trained visual-language model and querying the dictionary using clustering and semantic reasoning mechanisms to match material categories, it can improve the model's cross-material generalization ability for unseen materials. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the object physical property prediction method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the object physical property prediction method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the object physical property prediction method of this application; Figure 4 This is a schematic diagram of the module structure of the object physical property prediction method device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the object physical property prediction method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: inputting multi-view images into a pre-trained visual-language model to obtain a physical attribute dictionary; reconstructing 3DGS based on the multi-view images and combining a preset loss mechanism to obtain a three-dimensional Gaussian point cloud representation; embedding region-aware features into the three-dimensional Gaussian point cloud representation and performing joint supervised training on the three-dimensional Gaussian point cloud representation to obtain semantic region-aware features; clustering and semantic reasoning of Gaussian point clouds based on semantic region-aware features, querying the physical attribute dictionary to obtain the physical attribute value corresponding to each Gaussian point cloud.

[0025] In this embodiment, for ease of description, the following description uses the object physical attribute prediction system as the execution subject.

[0026] Because existing object reconstruction methods based on neural radiation fields do not have the ability to segment boundaries in the clip feature maps of 2D images, although they can basically correctly infer the approximate location at the semantic level, their segmentation effect is very poor. For objects reconstructed based on 3DGS, the 3D reconstruction results are prone to geometric collapse, surface holes or floating artifacts for objects with weak textures or simple geometric structures. Furthermore, due to the chaotic masks automatically segmented by the segmentation model and the lack of semantics, inference errors may occur.

[0027] This application provides a solution that effectively alleviates surface voids and geometric collapse by introducing a geometrically constrained enhanced 3DGS reconstruction mechanism. It endows point cloud component-level semantic discrimination capabilities by embedding region-aware features into Gaussian point clouds and jointly training them using multi-source supervision signals (including segmentation masks, semantic features, and structural features). Furthermore, it constructs a scalable physical property dictionary based on a pre-trained vision-language model and combines clustering and zero-shot feature matching to achieve accurate inference of physical properties for unseen materials. Therefore, this solution can simultaneously ensure high-quality geometric reconstruction and high-precision point cloud-level physical property prediction without material annotation, significantly improving the model's general physical perception capability in open-world scenarios.

[0028] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following description uses a personal computer as an example to illustrate this embodiment and the subsequent embodiments.

[0029] Based on this, embodiments of this application provide a method for predicting the physical properties of an object, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the object physical property prediction method of this application.

[0030] In this embodiment, the object physical property prediction method includes steps S10 to S50: Step S10: Obtain multi-view images of the target object; It should be noted that the multi-view images refer to RGB images of the target object taken from different perspectives, such as standard formats like JPEG and PNG; these images can be acquired using ordinary cameras, smartphones, industrial cameras, or other conventional image acquisition devices.

[0031] Step S20: Input the multi-view image into a pre-trained visual-language model to obtain a physical attribute dictionary; It should be noted that the Vision-Language Model (VLM) refers to a multimodal neural network model that is pre-trained on large-scale image and text data, capable of understanding image content and aligning it with natural language semantics, such as CLIP and ALIGN.

[0032] The physical property dictionary refers to a mapping table consisting of material categories and their corresponding physical property values, where physical property values ​​include, but are not limited to, parameters related to the mechanical or physical behavior of materials such as density, Young's modulus, Poisson's ratio, and thermal conductivity.

[0033] Understandably, since the material type of target objects in real-world scenarios is often unknown and difficult to label, relying on manual labeling or dedicated sensors to obtain physical properties would significantly increase costs and limit the versatility of the method. Therefore, step S20 utilizes the zero-shot reasoning capability of a pre-trained vision-language model to automatically infer possible material categories and their physical properties from multi-view images, avoiding reliance on large amounts of labeled data or dedicated measurement equipment, thereby achieving the ability to predict physical properties across materials and objects in an open world.

[0034] In this embodiment, the multi-view image of the target object is used as input and fed into a pre-trained vision-language model. Based on the matching degree between the image semantics and the preset material text description (such as "rubber", "metal", "wood" etc.), the model predicts several candidate material categories and, combined with a prior knowledge base or material database, generates a physical property dictionary containing material categories and their corresponding physical property values ​​(such as density, Young's modulus) for subsequent steps to query.

[0035] Step S30: Based on the multi-view image and combined with a preset loss mechanism, the three-dimensional Gaussian splash 3DGS is reconstructed to obtain a three-dimensional Gaussian point cloud representation. It should be noted that the 3D Gaussian Splatting (3DGS) is a neural rendering method based on explicit point cloud representation. It models the scene using a set of 3D Gaussian ellipsoids with attributes such as position, covariance, opacity and spherical harmonic coefficients, and uses rasterization to project the Gaussian point cloud onto the image plane to generate a differentiable 2D rendered image.

[0036] The aforementioned 3D Gaussian point cloud representation refers to an explicit 3D scene representation composed of the aforementioned Gaussian ellipsoid set. Each Gaussian point carries geometric and appearance information and can be used to efficiently and effectively render images from new perspectives.

[0037] Understandably, in standard 3DGS reconstruction frameworks, multi-view real images are typically used as supervision signals to optimize Gaussian point cloud parameters by minimizing the L1 photometric loss between the rendered and real images. However, this strategy primarily focuses on color fidelity and lacks explicit constraints on geometry, which can easily lead to problems such as surface holes, geometric collapse, or topological errors in the reconstruction results, making it difficult to recover detailed and consistent object geometry.

[0038] Therefore, this embodiment introduces a preset loss mechanism in addition to the basic photometric loss to enhance the constraints on the 3DGS geometric reconstruction process. This loss mechanism includes, but is not limited to, geometrically aware terms such as boundary consistency loss and normal smoothing loss, which can effectively guide Gaussian point clouds to maintain reasonable surface continuity and structural integrity during the optimization process, thereby obtaining a more geometrically high-quality and robust 3D Gaussian point cloud representation.

[0039] In one feasible embodiment, step S30 may include steps S31 to S34: Step S31: Calculate the boundary-aware loss function between the rendered transparent image corresponding to the multi-view image and the mask of the real object; It should be noted that the boundary-aware loss function is used to ensure accurate capture of object edge information during the 3D model reconstruction process. By comparing the edge regions in the rendered image and the real image, the difference between the two is minimized to enhance the accuracy of the reconstructed model boundary.

[0040] In this embodiment, when rendering a 3D Gaussian point cloud into a 2D image using rasterization, a corresponding transparency map (i.e., an α map) is simultaneously generated, representing the degree of opacity at each pixel. The rendered transparency map is then compared pixel-by-pixel with the real object mask extracted from the multi-view image, and a boundary-aware loss function is calculated between the two. Specifically, this loss function measures the consistency between the rendered transparency map and the real mask in the edge region, and is defined as follows:

[0041] In the formula, It is a rasterized rendering transparency image. It is the corresponding real object mask image, where H and W represent the height and width of the image, respectively, and N is the total number of pixels.

[0042] Minimizing this loss can effectively enhance the reconstruction model's ability to accurately recover object boundaries, reduce floating point clouds, and improve geometric integrity.

[0043] Step S32: Calculate the local gradient of the rendering depth map corresponding to the multi-view image to obtain the smoothing loss function; It should be noted that the smoothing loss function is used to ensure the continuity and smoothness of the 3D Gaussian point cloud surface. By performing a consistency check on the normal vectors of adjacent Gaussian points and applying smoothing constraints, the reconstruction result can present a more natural and abrupt surface morphology, reducing the influence of noise and irregular shapes.

[0044] In this embodiment, based on the depth map rendered from a 3D Gaussian point cloud, the local gradient difference in the horizontal and vertical directions is calculated to construct a smoothing loss function. This loss function is used to constrain the spatial continuity of the depth map and avoid drastic surface fluctuations or noise. Specifically, the smoothing loss function is defined as:

[0045] In the formula, This represents the value of pixel (i,j) in the rendered 2D depth map, where H and W represent the height and width of the image, respectively, and N is the total number of pixels.

[0046] By minimizing this loss, the depth map can be made to maintain a smooth transition in space, reducing holes in object reconstruction and thus improving the continuity and stability of surface geometry.

[0047] Step S33: Based on the normalized image gradient and corresponding depth information of each pixel in the multi-view image, estimate the surface normal in the image space, and combine it with the geometric normal calculated by the three-dimensional Gaussian point cloud to construct the normal vector. It should be noted that the normal vector is used to describe the local surface orientation of each Gaussian point. It not only helps determine the relative positional relationships between points but also serves as a key parameter in calculating the normal smoothing loss to achieve the aforementioned smoothing constraints. Furthermore, the normal vector is crucial for advanced rendering techniques such as ray tracing and shadow generation, thereby improving the realism of the final rendered image.

[0048] In this embodiment, the normalized image gradient of each pixel in the multi-view image (i.e., The surface normal (N(p)) in the image space is estimated using a local plane fitting method, taking into account the corresponding depth information. Specifically, a plane is constructed by selecting three neighboring pixels, and the normal vector of this plane is then calculated as the local surface normal.

[0049] Simultaneously, from the 3D Gaussian point cloud, based on the position and covariance matrix of the Gaussian points, the corresponding geometric normals (N(p)) are obtained through geometric calculation. Finally, the normals estimated from the image space ( The geometric normals (N(p)) of the Gaussian point cloud are compared with those of the real surface normals to construct a normal consistency constraint term, which guides the Gaussian point cloud to maintain consistency with the real surface normals during the optimization process. The definition of the normal consistency constraint term is:

[0050] In the formula, I represents the images in the training set, and W(I) represents the number of pixels in each image. I(p) is the normalized image gradient of each pixel in image I, Nd(p) is the surface normal calculated from the depth information, that is, a plane is constructed by three point clouds and then the surface normal is calculated, and N(p) is the normal estimated from the image geometry.

[0051] Step S34: Combine the boundary-aware loss function, the smoothing loss function, and the normal vector to apply geometric constraints to the reconstruction process of the 3D Gaussian splash, and construct the 3D Gaussian point cloud representation.

[0052] Understandably, since the standard 3D Gaussian splatter (3DGS) reconstruction method mainly relies on photometric consistency loss (such as L1 loss) for optimization, it lacks explicit constraints on the geometric structure, which easily leads to problems such as blurred boundaries, discontinuous normals, surface voids or geometric collapse in complex surface areas. Therefore, performing step S31 can further optimize the geometric structure of the 3D Gaussian splatter model in addition to the basic photometric loss, ensuring that the reconstructed 3D Gaussian point cloud representation not only has good appearance fidelity, but also has accurate geometric features and high-quality surface details.

[0053] In another feasible embodiment, S34 may further include steps S341 to S342. Step S311: Summing the boundary-aware loss, the smoothing loss function, and the normal vector yields the total geometric reconstruction loss function; In this embodiment, the three types of geometric constraint losses are weighted and combined to form the total geometric reconstruction loss function. Specifically, the total loss function is defined as:

[0054] in, It is a smoothing loss function. It is a boundary-aware loss function. This refers to the normal consistency constraint term obtained from the normal vector. It is the standard 3DGS photometric loss. The weighting coefficient for smoothing the loss is set to 0.2 to achieve stable optimization.

[0055] Step S312: Apply geometric constraints to the reconstruction process of the three-dimensional Gaussian splash using the total geometric reconstruction loss function to construct the three-dimensional Gaussian point cloud representation.

[0056] In this embodiment, the aforementioned total geometric reconstruction loss function is used as an additional supervision signal, participating together with the original photometric loss function in the backpropagation optimization process of the 3D Gaussian Splash (3DGS) model. By jointly minimizing the photometric and geometric losses, the Gaussian point cloud not only pursues color matching during the optimization process but also takes into account the rationality and integrity of the geometric structure, ultimately constructing a 3D Gaussian point cloud representation with high-quality geometric features.

[0057] By introducing boundary-aware loss, smoothing loss, and normal consistency constraints through the above steps, the shortcomings of the standard 3DGS method in geometric reconstruction are effectively compensated, and fine control over object boundaries, surface continuity, and local geometric structures is achieved, which significantly improves the quality and robustness of 3D reconstruction.

[0058] Step S40: Embed region-aware features into the three-dimensional Gaussian point cloud representation, and perform joint supervised training on the three-dimensional Gaussian point cloud representation to obtain semantic region-aware features; It should be noted that the region-aware feature refers to a learnable high-dimensional latent vector (e.g., 32-dimensional) attached to each three-dimensional Gaussian point. This vector can be rendered into a feature map through rasterization and can encode local geometric and semantic information to distinguish different parts or semantic regions of an object.

[0059] The semantic perception features refer to the dual capabilities of the region perception features after joint supervised training: on the one hand, they can reflect the boundaries of component-level regions (such as "handle", "cup body", "wheels" etc.), and on the other hand, they can align with high-level semantic concepts (such as material category, functional attributes), thereby supporting subsequent clustering and physical attribute reasoning.

[0060] Understandably, since standard 3D Gaussian splashing only models color and geometry, its point cloud itself lacks semantic discrimination capabilities. If used directly for physical attribute assignment, it is easy to assign the same attributes to different parts of the same object (such as predicting "metal cup body" and "plastic cup lid" as metal), causing errors in physical reasoning. Therefore, step S40 is executed to jointly optimize the region-aware features by introducing multi-source supervision signals—including the part-level segmentation mask generated by the SAM model, CLIP semantic features extracted from multi-view images, and DINOv2 structural features. This avoids semantic ambiguity and attribute misassignment problems, thereby achieving fine-grained semantic perception and cross-modal semantic alignment capabilities at the point cloud level, laying the foundation for subsequent high-precision physical attribute prediction.

[0061] In one feasible embodiment, step S40 may include steps S41 to S45: Step S41: Embed the region-aware features into each 3D Gaussian point cloud, and convert the region-aware features into a target dimension feature map through rasterization. In this embodiment, based on the geometrically reconstructed 3D Gaussian point cloud, an additional learnable 32-dimensional one-dimensional vector is added to each Gaussian point as a region-aware feature. This vector is independent of the original 3DGS attributes such as position, covariance, spherical harmonics, and opacity; during training, the original geometric and appearance parameters of the 3DGS are frozen, and only this 32-dimensional vector is allowed to be optimized.

[0062] Subsequently, using the rasterization rendering mechanism of 3DGS, the 32-dimensional vector of each Gaussian point is projected and accumulated onto the image plane to generate a 32-channel target dimension feature map (instead of a traditional 3-channel RGB image). This feature map preserves the spatial correspondence and can be used for subsequent supervised training.

[0063] Step S42: Divide the target dimension feature map into a first sub-dimensional feature map and a second sub-dimensional feature map; In this embodiment, the 32-dimensional feature map is divided into two sub-feature maps according to the channel dimension: First sub-dimensional feature map: Take the first 16 channels to learn the component-level region structure information of the object; Second sub-dimensional feature map: Take the last 16 channels, denoted as... It is used to learn cross-modal semantics and structural representations.

[0064] This partitioning method allows different semantic objectives to be optimized separately through independent supervision signals, achieving functional decoupling.

[0065] Step S43: For the first sub-dimensional feature map, the multi-scale segmentation mask extracted from the multi-view image by the pre-trained SAM model for segmenting everything is used as the first supervision target. In this embodiment, the same multi-view RGB image used for 3D reconstruction is input into a pre-trained SAM model, which automatically generates multi-scale, class-unlabeled segmentation masks. These masks reflect the boundaries of potential semantic parts in the image (such as "cup body", "handle", "wheel", etc.).

[0066] These SAM masks are used as the first supervision target to guide the learning of the first sub-dimensional feature map. The purpose is to make Gaussian points belonging to the same mask region tend to be similar in the feature space, while the Gaussian point features between different mask regions have obvious differences, thereby giving the point cloud component-level region perception capability.

[0067] Step S44: For the second sub-dimensional feature map, the semantic feature map and structural feature map extracted from the multi-view image are used as the second supervision target; In this embodiment, pre-trained visual-language models (such as CLIP) and self-supervised visual models (such as DINOv2) are used to extract the corresponding feature maps from the multi-view RGB images.

[0068] Among them, CLIP semantic feature map: represents the alignment information between image content and text semantics; DINOv2 structural feature map: captures local geometry and component layout information.

[0069] These two types of feature maps are used as the second supervision target. Specifically, the second sub-dimensional feature map... Input a lightweight mapping network (e.g., a two-layer fully connected network), output the predicted CLIP feature map. and DINOv2 feature map The points are then aligned with the extracted CLIP and DINOv2 feature maps. This process aims to embed each Gaussian point into a feature representation that can be matched with textual or structural semantics, thereby supporting text-based point cloud localization and zero-shot semantic retrieval.

[0070] Step S45: Combine the first supervised target, the second supervised target, the loss function corresponding to the first supervised target, and the loss function of the second supervised target to train the three-dimensional Gaussian point cloud to obtain semantic region-aware features.

[0071] In this embodiment, while keeping the 3DGS ensemble parameters frozen, only the 32-dimensional region-aware feature vector of each Gaussian point is optimized. This is achieved through joint optimization: a region consistency constraint between the first sub-dimensional feature map and the SAM mask; and a semantic alignment constraint between the mapped second sub-dimensional feature map and the CLIP / DINOv2 feature map. The resulting 32-dimensional feature is the semantic region-aware feature—its first 16 dimensions encode component region attribution, and its last 16 dimensions encode semantic embeddings that can be queried via text. Both dimensions jointly support subsequent clustering, material identification, and physical property assignment.

[0072] Through the above steps, it is possible to embed semantic region-aware features with both region discrimination and cross-modal semantic alignment capabilities into Gaussian point clouds while maintaining the three-dimensional geometric structure, thereby supporting fine-grained component-level physical attribute prediction and text-queryable semantic localization.

[0073] Step S50: Based on the semantic region perception features, cluster and semantic reasoning are performed on the Gaussian point clouds, and the physical attribute dictionary is queried to obtain the physical attribute value corresponding to each Gaussian point cloud.

[0074] It should be noted that the physical property values ​​include, but are not limited to, parameters related to the physical behavior of an object, such as material type, density, Young's modulus, Poisson's ratio, thermal conductivity, and coefficient of friction.

[0075] Understandably, since individual Gaussian points lack a clear semantic identity, directly matching physical attributes point by point is susceptible to noise or feature drift, leading to inconsistent or unreasonable attribute allocation. Therefore, step S50 is executed to first cluster the Gaussian point cloud based on semantic region perception features to form semantically consistent component regions. Then, semantic reasoning is performed using the cluster centers as representatives, and the physical attribute dictionary is queried. This avoids the instability and semantic fragmentation problems of point-level prediction, thereby achieving component-level consistent and semantically interpretable physical attribute allocation, significantly improving the rationality and robustness of the prediction results.

[0076] In one feasible embodiment, step S50 may include steps S51 to S55: Step S51: The semantic region-aware features are fused with the Gaussian point cloud to obtain latent features; In this embodiment, after the aforementioned joint supervised training, each 3D Gaussian point cloud possesses a 32-dimensional semantic region-aware feature. To enhance geometric-semantic consistency during clustering, this semantic region-aware feature is weighted and fused with the location information of the Gaussian point cloud to form a new latent feature vector z. Specifically, the location information is added to the semantic feature with a small weight (e.g., 0.1) to ensure that spatial proximity is preserved while prioritizing semantic consistency during clustering. This latent feature z serves as the basic input for subsequent clustering and inference.

[0077] Step S52: A density-based noise spatial clustering algorithm is used to embed the location information of the Gaussian point cloud into the latent features for clustering to obtain cluster centers; In this embodiment, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is used to cluster the fused latent features z. This algorithm automatically identifies high-density clusters by analyzing the local density distribution of data points and marks points in low-density areas as noise.

[0078] In this process, the positional information of the Gaussian point cloud is explicitly embedded into the latent features to enhance spatial coherence. This allows point clouds belonging to the same physical component (such as "cup body" or "handle") to be grouped into the same cluster due to their spatial proximity, even if they differ slightly in semantic features. The final output consists of multiple cluster centers, representing different semantic components of the object.

[0079] Step S53: Input the semantic region-aware features of the cluster center into the pre-trained mapping model to generate corresponding semantic features and structural features; In this embodiment, the average semantic region-aware features (i.e., the last 16-dimensional vector) of the Gaussian points corresponding to each cluster center are extracted and input into a pre-trained lightweight mapping network model. This model, after training, can map 32-dimensional latent features to two independent feature spaces: CLIP semantic features : Reflects high-level semantic information such as material category; DINOv2 structural features Capture local geometry and component layout information.

[0080] Through the above steps, the transformation from point cloud latent representation to cross-modal semantic and structural representation is realized, providing support for subsequent zero-shot inference.

[0081] Step S54: Calculate the cosine similarity between the semantic features and the semantic features of the preset material category text description, and determine the material category corresponding to the cluster center based on the maximum similarity; In this embodiment, a set of material category text descriptions (such as "metal", "plastic", "wood", "rubber", etc.) are pre-constructed and encoded into corresponding textual semantic features using a pre-trained CLIP model. CLIP features are generated for each cluster center. The cosine similarity between the cluster center and the textual features of all materials is calculated. The material category with the highest similarity is selected as the predicted result of the cluster center, thus achieving zero-shot material classification.

[0082] Step S55: Based on the material category, query the corresponding physical property value in the physical property dictionary.

[0083] In this embodiment, after determining the material category of a cluster center, the corresponding physical attribute values ​​are directly retrieved from the physical attribute dictionary generated in step S20, and these attribute values ​​are assigned to all Gaussian point clouds contained in that cluster center. This achieves point cloud-level physical attribute annotation for all components of the entire object.

[0084] The above-described method involves inputting multi-view images into a pre-trained visual-language model to obtain a physical attribute dictionary. Based on the multi-view images and a pre-defined loss mechanism, 3DGS is reconstructed to obtain a 3D Gaussian point cloud representation. Region-aware features are embedded into the 3D Gaussian point cloud representation, and joint supervised training is performed to obtain semantic region-aware features. Based on the semantic region-aware features, clustering and semantic reasoning are performed on the Gaussian point clouds, and the physical attribute dictionary is queried to obtain the physical attribute value corresponding to each Gaussian point cloud. This scheme achieves integrated and collaborative optimization of geometric reconstruction, semantic understanding, and physical attribute prediction. Without the need for manual annotation and dedicated sensors, it significantly improves the accuracy, semantic consistency, and cross-material generalization ability of point cloud-level physical attribute estimation, providing an end-to-end, zero-shot, and scalable technical path for general 3D physical perception.

[0085] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S45, the object physical property prediction method further includes steps A11 to A15: Step A11: Calculate the feature similarity between any two pixels in the first sub-dimension feature map; In this embodiment, for the first sub-dimensional feature map F generated by rasterizing the 3D Gaussian point cloud, all pixel pairs in the image are traversed ( , ), calculate its corresponding eigenvector F( ) and F( The cosine similarity between two terms serves as the basis for measuring their semantic relevance. Specifically, the cosine similarity function is defined as:

[0086] in, F( ), F( ) This represents the vector inner product, used to reflect the proximity of two pixel features in space. This similarity value will serve as the basis for subsequently constructing consistency and discriminative constraints.

[0087] Step A12: Determine whether the pixel belongs to the same semantic region defined by the multi-scale segmentation mask; In this embodiment, a pre-trained SAM model is used to extract segmentation masks of multiple scales from multi-view images. Each mask corresponds to an independent semantic region (such as "cup body" or "handle"). For each pair of pixels ( , ), query whether it is located in the same mask region, and get the binary label coor( )∈{0,1}: If coor( )=1 indicates that both belong to the same semantic region; If coor( =0 indicates that the two belong to different regions.

[0088] This judgment result is used to guide the construction direction of subsequent loss terms.

[0089] Step A13: If yes, then construct a consistency loss term based on the feature similarity to constrain pixel features in the same semantic region to tend to be similar. When determining pixels and When they belong to the same semantic region (i.e., coor( () = 1), expecting them to have high consistency in the feature space. To this end, a consistency loss term is constructed. ( ), its form is: ,0] When coor( When ) = 1, the above simplifies to max[ At this point, the loss approaches zero only when... The loss tends to 1, meaning the more similar the features are, the smaller the loss, thus forcing pixels in the same region to become consistent at the feature level.

[0090] Step A14: If not, construct a discriminative loss term based on the feature similarity to constrain the differences in pixel feature regions of different semantic regions; When the judgment and When they do not belong to the same semantic region (i.e., coor( When the sum of its features is 0, excessive similarity should be avoided. In this case, the loss term becomes: ,0]=

[0091] That is, a negative loss value means that the optimization objective is to make coordinator (coordinator) ... To minimize the size of the semantic regions, thereby enhancing the feature differences between different semantic regions and clarifying semantic boundaries.

[0092] Step A15: The consistency loss term and the discriminative loss term are merged into a region-aware loss function, and the region-aware loss function is used as the loss function of the first supervised target.

[0093] In this embodiment, all pixel pairs ( ) corresponding Summing these values ​​yields the overall region-aware loss function:

[0094] In the formula, W(I) is the set of all pixels in image I, H and W are the height and width of the feature map, and (1-||F(p)||2) calculates the feature normalization loss term for each pixel, encouraging the feature vector to approach the unit norm. It is a consistent loss item.

[0095] The method described above achieves refined supervision of the first sub-dimensional feature map by constructing a dynamic loss function based on pixel pair relationships. This method not only improves the regional consistency of the feature map but also enhances the clarity of semantic boundaries, providing a high-quality region-aware foundation for subsequent clustering and physical attribute reasoning, significantly outperforming traditional unsupervised or single-supervised methods.

[0096] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Before step S45, the object physical property prediction method further includes steps B11 to B13: Step B11: Calculate the cosine similarity between the semantic feature map and the structural feature map; In this embodiment, the second sub-dimensional feature map generated by rasterizing the 3D Gaussian point cloud is converted into two output feature maps using a pre-trained lightweight mapping model: the CLIP feature map. and DINOv2 feature map .

[0097] Simultaneously, the ground reference feature map extracted from the corresponding multi-view RGB image can be set as the image-level semantic feature map extracted by the CLIP model. and image-level structural feature maps extracted from the DINOv2 model. Subsequently, the cosine similarity between the predicted feature map and the true feature map is calculated as an indicator to measure the degree of semantic and structural consistency.

[0098] Step B12: Construct a semantic alignment loss term based on the cosine similarity; Because relying solely on single-modal features (such as using only CLIP semantic features or only DINOv2 structural features) makes it difficult to simultaneously consider both the high-level semantic category and local geometric structure of an object, it can easily lead to biases in point cloud features during semantic reasoning, such as "having semantics but no structure" or "having structure but no semantics." Therefore, this embodiment calculates the cosine similarity between the semantic feature map and the structural feature map, and uses this to construct a semantic alignment loss term. This guides the model to collaboratively optimize the consistency of the expression of the two features during training, so that the latent features embedded in each Gaussian point can both respond to textual semantic queries and maintain alignment with the structure of the real object. This provides a more robust and reliable multimodal semantic foundation for subsequent zero-shot physical property prediction.

[0099] Specifically, this embodiment calculates the pixel-wise cosine similarity between the CLIP semantic feature map output by the mapping model and the DINOv2 structural feature map at their corresponding spatial locations, and constructs a loss term based on this similarity: the loss is small when the semantic features and structural features are expressed consistently in the same region (i.e., high similarity); otherwise, the loss increases. By minimizing this loss, the model is explicitly constrained to learn a joint representation that supports the semantics and structure, avoiding decoupling or conflict between the two modal features during the optimization process, thereby ensuring that the Gaussian point cloud can accurately reflect the true geometry and component layout of the object while possessing text query capabilities.

[0100] Step B13: Use the semantic alignment term as the loss function of the second supervision target.

[0101] In this embodiment, the semantic alignment loss term constructed above is used as the loss function of the second supervision objective to guide the second sub-dimensional feature map. The optimization process involves minimizing this loss so that the latent features of each Gaussian point can not only accurately reflect its local geometric structure, but also embed high-level information that can be matched with text semantics, ultimately achieving point cloud-level semantic interpretability and structure awareness.

[0102] Using the methods described above, a loss function for the second supervised objective is generated based on the cosine similarity between the CLIP semantic feature map extracted from multi-view images and the DINOv2 structural feature map. This allows the latent features embedded in the Gaussian point cloud to be aligned with high-level semantic and local structural information during training, effectively avoiding the problem of semantic and geometric representation being disconnected. This improves the accuracy and robustness of point clouds in zero-shot material recognition and physical property reasoning.

[0103] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the object physical property prediction method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0104] This application also provides an object physical property prediction device, please refer to... Figure 4 The object physical property prediction device includes: Image acquisition module 10 is used to acquire multi-view images of the target object; Image processing module 20 is used to input the multi-view image into a pre-trained visual-language model to obtain a physical attribute dictionary; The reconstruction module 30 is used to reconstruct the three-dimensional Gaussian splash 3DGS based on the multi-view image and in combination with a preset loss mechanism to obtain a three-dimensional Gaussian point cloud representation. Training module 40 is used to embed region-aware features into the three-dimensional Gaussian point cloud representation and perform joint supervised training on the three-dimensional Gaussian point cloud representation to obtain semantic region-aware features. The reasoning module 50 is used to perform clustering and semantic reasoning on Gaussian point clouds based on the semantic region perception features, query the physical attribute dictionary, and obtain the physical attribute value corresponding to each Gaussian point cloud.

[0105] The object physical property prediction method and apparatus provided in this application, employing the object physical property prediction method in the above embodiments, can solve the technical problem of how to improve the semantic accuracy and cross-material generalization ability of point cloud-level physical property prediction while ensuring the quality of 3D geometric reconstruction. Compared with the prior art, the beneficial effects of the object physical property prediction method and apparatus provided in this application are the same as those of the object physical property prediction method provided in the above embodiments, and other technical features in the object physical property prediction method and apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0106] This application provides an apparatus for predicting the physical properties of an object. The apparatus includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the physical property prediction method of the above embodiment.

[0107] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an apparatus suitable for implementing the object physical property prediction method in the embodiments of this application. The object physical property prediction method apparatus in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The illustrated object physical property prediction method and device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0108] like Figure 5As shown, the object physical property prediction method device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the object physical property prediction method device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the object physical property prediction method apparatus to communicate wirelessly or wiredly with other devices to exchange data. Although the object physical property prediction method apparatus with various systems is shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0109] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0110] The object physical property prediction method and device provided in this application, employing the object physical property prediction method in the above embodiments, can solve the technical problem of how to improve the semantic accuracy and cross-material generalization ability of point cloud-level physical property prediction while ensuring the quality of 3D geometric reconstruction. Compared with the prior art, the beneficial effects of the object physical property prediction method and device provided in this application are the same as those of the object physical property prediction method provided in the above embodiments, and other technical features in this object physical property prediction method and device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0111] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0113] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the object physical property prediction method in the above embodiments.

[0114] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0115] The aforementioned computer-readable storage medium may be included in the object physical property prediction method device; or it may exist independently and not assembled into the object physical property prediction method device.

[0116] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the object physical attribute prediction method device, the object physical attribute prediction method device: acquires multi-view images of the target object; inputs the multi-view images into a pre-trained visual-language model to obtain a physical attribute dictionary; reconstructs 3DGS based on the multi-view images and in conjunction with a preset loss mechanism to obtain a 3D Gaussian point cloud representation; embeds region-aware features into the 3D Gaussian point cloud representation and performs joint supervised training on the 3D Gaussian point cloud representation to obtain semantic region-aware features; performs clustering and semantic reasoning on the Gaussian point clouds based on the semantic region-aware features, queries the physical attribute dictionary, and obtains the physical attribute value corresponding to each Gaussian point cloud.

[0117] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0120] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described object physical property prediction method. This solves the technical problem of improving the semantic accuracy and cross-material generalization ability of point cloud-level physical property prediction while ensuring the quality of 3D geometric reconstruction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the object physical property prediction method provided in the above embodiments, and will not be repeated here.

[0121] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the object physical property prediction method described above.

[0122] The computer program product provided in this application solves the technical problem of improving the semantic accuracy and cross-material generalization ability of point cloud-level physical property prediction while ensuring the quality of 3D geometric reconstruction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the object physical property prediction method provided in the above embodiments, and will not be repeated here.

[0123] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for predicting the physical properties of an object, characterized in that, The method for predicting the physical properties of an object includes: Obtain multi-view images of the target object; The multi-view images are input into a pre-trained visual-language model to obtain a physical attribute dictionary; Based on the multi-view images and combined with a preset loss mechanism, the 3D Gaussian splash 3DGS is reconstructed to obtain a 3D Gaussian point cloud representation. Region-aware features are embedded in the 3D Gaussian point cloud representation, and joint supervised training is performed on the 3D Gaussian point cloud representation to obtain semantic region-aware features. Based on the semantic region perception features, clustering and semantic reasoning are performed on Gaussian point clouds, and the physical attribute dictionary is queried to obtain the physical attribute value corresponding to each Gaussian point cloud.

2. The method for predicting the physical properties of an object as described in claim 1, characterized in that, The step of reconstructing a 3D Gaussian splash 3DGS based on the multi-view images and in conjunction with a preset loss mechanism to obtain a 3D Gaussian point cloud representation includes: Calculate the boundary-aware loss function between the rendered transparent image corresponding to the multi-view image and the mask of the real object; Calculate the local gradient of the rendering depth map corresponding to the multi-view image to obtain the smoothing loss function; Based on the normalized image gradient and corresponding depth information of each pixel in the multi-view image, the surface normal in the image space is estimated, and combined with the geometric normal calculated by the three-dimensional Gaussian point cloud, a normal vector is constructed. By combining the boundary-aware loss function, the smoothing loss function, and the normal vector, geometric constraints are applied to the reconstruction process of the 3D Gaussian splash, and the 3D Gaussian point cloud representation is constructed.

3. The object physical property prediction method as described in claim 2, characterized in that, The steps of combining the boundary-aware loss function, the smoothing loss function, and the normal vector to geometrically constrain the reconstruction process of the 3D Gaussian splash and construct the 3D Gaussian point cloud representation include: The boundary-aware loss, the smoothing loss function, and the normal vector are summed to obtain the total geometric reconstruction loss function; The reconstruction process of the 3D Gaussian splash is geometrically constrained by the total geometric reconstruction loss function to construct the 3D Gaussian point cloud representation.

4. The method for predicting the physical properties of an object as described in claim 1, characterized in that, The steps of embedding region-aware features into the 3D Gaussian point cloud representation and performing joint supervised training on the 3D Gaussian point cloud representation to obtain semantic region-aware features include: The region-aware features are embedded into each 3D Gaussian point cloud, and the region-aware features are converted into a target dimension feature map through rasterization. The target dimension feature map is divided into a first sub-dimensional feature map and a second sub-dimensional feature map; For the first sub-dimensional feature map, the multi-scale segmentation mask extracted from the multi-view image by the pre-trained SAM model for segmenting everything is used as the first supervision target; For the second sub-dimensional feature map, the semantic feature map and structural feature map extracted from the multi-view image are used as the second supervision target; By combining the first supervisory target, the second supervisory target, the loss function corresponding to the first supervisory target, and the loss function of the second supervisory target, the semantic region-aware features are obtained by training the three-dimensional Gaussian point cloud.

5. The method for predicting the physical properties of an object as described in claim 4, characterized in that, Before the step of training the 3D Gaussian point cloud to obtain semantic region-aware features by jointly using the first supervised target, the second supervised target, the loss function corresponding to the first supervised target, and the loss function of the second supervised target, the method further includes: Calculate the feature similarity between any two pixels in the first sub-dimensional feature map; Determine whether the pixel belongs to the same semantic region defined by the multi-scale segmentation mask; If so, a consistency loss term is constructed based on the feature similarity to constrain pixel features in the same semantic region to tend to be similar; If not, a discriminative loss term is constructed based on the feature similarity to constrain the differences in pixel feature regions of different semantic regions; The consistency loss term and the discriminative loss term are combined into a region-aware loss function, and the region-aware loss function is used as the loss function for the first supervised target.

6. The method for predicting the physical properties of an object as described in claim 4, characterized in that, Before the step of training the 3D Gaussian point cloud to obtain semantic region-aware features by jointly using the first supervised target, the second supervised target, the loss function corresponding to the first supervised target, and the loss function of the second supervised target, the method further includes: Calculate the cosine similarity between the semantic feature map and the structural feature map; A semantic alignment loss term is constructed based on the cosine similarity; The semantic alignment term is used as the loss function for the second supervision objective.

7. The method for predicting the physical properties of an object as described in claim 1, characterized in that, The steps of clustering and semantically reasoning Gaussian point clouds based on the semantic region-aware features, and querying the physical attribute dictionary to obtain the physical attribute value corresponding to each Gaussian point cloud include: The semantic region-aware features are fused with the Gaussian point cloud to obtain latent features; A density-based noise-based spatial clustering algorithm is used to embed the location information of the Gaussian point cloud into the latent features for clustering, thereby obtaining cluster centers; The semantic region-aware features of the cluster centers are input into a pre-trained mapping model to generate corresponding semantic and structural features; Calculate the cosine similarity between the semantic features and the semantic features of the preset material category text description, and determine the material category corresponding to the cluster center based on the maximum similarity; Based on the material category, the corresponding physical property value is queried from the physical property dictionary.

8. A device for predicting the physical properties of an object, characterized in that, The object physical property prediction device includes: The image acquisition module is used to acquire multi-view images of the target object; The image processing module is used to input the multi-view images into a pre-trained visual-language model to obtain a physical attribute dictionary; The reconstruction module is used to reconstruct the three-dimensional Gaussian splash 3DGS based on the multi-view image and in combination with a preset loss mechanism to obtain a three-dimensional Gaussian point cloud representation. The training module is used to embed region-aware features into the 3D Gaussian point cloud representation and perform joint supervised training on the 3D Gaussian point cloud representation to obtain semantic region-aware features. The reasoning module is used to perform clustering and semantic reasoning on Gaussian point clouds based on the semantic region perception features, and to query the physical attribute dictionary to obtain the physical attribute value corresponding to each Gaussian point cloud.

9. A device for predicting the physical properties of an object, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the object physical property prediction method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the object physical property prediction method as described in any one of claims 1 to 7.

Images