A three-dimensional model image processing method, system, terminal and storage medium based on semantic segmentation and point sampling

Abstract

The application relates to the technical field of computer graphics, and discloses a three-dimensional model image processing method, system, terminal and storage medium based on semantic segmentation and point sampling, the method comprising the following steps: obtaining a three-dimensional grid model to be unfolded, performing semantic segmentation processing on the three-dimensional grid model to be unfolded through a semantic segmentation network to obtain a semantic three-dimensional grid model; obtaining a surface area ratio and a multilayer perception machine, performing point sampling on the semantic three-dimensional grid model according to the surface area ratio to obtain training data, inputting the training data into the multilayer perception machine for training to obtain a two-dimensional coordinate prediction model; obtaining a semantic part of the three-dimensional grid model to be unfolded, performing coordinate prediction on the semantic part through the two-dimensional coordinate prediction model to obtain a position unfolding diagram. The application divides parts based on semantic segmentation, samples a multilayer perception machine according to a surface area ratio, predicts position coordinates of each part, generates a position unfolding diagram corresponding to semantics, and realizes accurate semantic understanding of a three-dimensional model unfolding diagram.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to a method, system, terminal, and computer-readable storage medium for processing three-dimensional model images based on semantic segmentation and point sampling. Background Technology

[0002] In fields such as computer graphics, digital entertainment, industrial design, and virtual reality / augmented reality (VR / AR), parametric unwrapping of UV coordinates for 3D mesh models is a crucial preprocessing technique. UV coordinate unwrapping aims to map the vertices of a 3D model's surface to a 2D plane, forming a 2D coordinate representation called a UV coordinate unwrapped map. This unwrapped map forms the basis for subsequent downstream operations such as texture mapping, normal mapping, and material baking. A high-quality, low-distortion, highly readable, and easy-to-use UV coordinate unwrapped map can significantly improve the visual fidelity and modification efficiency of 3D models.

[0003] Traditional 3D model UV unwrapping methods (such as manual segmentation and flattening, and neural network-based global mapping) are highly dependent on manual intervention and have limited universality. They also lack an understanding of the semantic information at the model component level. As a result, although the generated UV coordinate unwrapped map may have low overall distortion, its UV coordinate division does not have a clear semantic correspondence. This makes it difficult to perform independent and refined texture editing and downstream processing on specific semantic components, and it is difficult to achieve accurate semantic understanding of the unwrapped map of the 3D model. This has become a problem that urgently needs to be solved.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a three-dimensional model image processing method, system, terminal, and computer-readable storage medium based on semantic segmentation and point sampling. This invention aims to solve the problem in the prior art that, due to its high dependence on manual intervention and limited universality, coupled with a lack of understanding of semantic information at the model component level, it is impossible to accurately identify the generated UV coordinate unfolded diagram, resulting in difficulty in accurately understanding the semantics of the unfolded diagram of the three-dimensional model.

[0006] To achieve the above objectives, the present invention provides a three-dimensional model image processing method based on semantic segmentation and point sampling, the method comprising the following steps: A 3D mesh model to be unfolded is obtained, and a semantic segmentation network is used to perform semantic segmentation processing on the 3D mesh model to be unfolded to obtain a semantic 3D mesh model. Obtain the surface area ratio, perform point sampling on the semantic 3D mesh model according to the surface area ratio to obtain training data, and train the corresponding multilayer perceptrons with the obtained training data to obtain a 2D coordinate prediction model for each semantic component. Obtain the semantic components of the three-dimensional mesh model to be unfolded, and predict the coordinates of the semantic components using the two-dimensional coordinate prediction model to obtain the position unfolding map.

[0007] Optionally, in the 3D model image processing method based on semantic segmentation and point sampling, the 3D mesh model to be unfolded includes vertex coordinates, patch indices, normal vectors, and texture coordinates. The process of obtaining the 3D mesh model to be unfolded, and performing semantic segmentation on the 3D mesh model using a semantic segmentation network to obtain a semantic 3D mesh model, specifically includes: Obtain vertex coordinates, patch indexes, normal vectors, and texture coordinates; normalize and unify the coordinates and scales of the vertex coordinates, patch indexes, normal vectors, and texture coordinates to obtain preprocessed data. The preprocessed data is semantically segmented using the encoder and decoder paths of the semantic segmentation network to obtain a semantic 3D mesh model.

[0008] Optionally, the 3D model image processing method based on semantic segmentation and point sampling, wherein the step of performing semantic segmentation processing on the preprocessed data through a semantic segmentation network to obtain a semantic 3D mesh model specifically includes: The preprocessed data is normalized according to the grid simplification algorithm to obtain a normalized grid; A three-dimensional mesh data set and cross-entropy loss are obtained. The normalized mesh is then semantically segmented based on the three-dimensional mesh data set and the cross-entropy loss through the encoder and decoder paths of the semantic segmentation network to obtain a semantic three-dimensional mesh model.

[0009] Optionally, the three-dimensional model image processing method based on semantic segmentation and point sampling, wherein obtaining the surface area ratio, performing point sampling on the semantic three-dimensional mesh model according to the surface area ratio to obtain training data, and training the corresponding multilayer perceptrons with the obtained training data to obtain two-dimensional coordinate prediction models for each semantic component, specifically includes: The surface area ratio is obtained, and based on the inverse transform sampling method, point sampling is performed on the semantic 3D mesh model according to the surface area ratio to obtain triangular patches. The triangular facets are selected using the centroid coordinate interpolation method to obtain the coordinates of multiple three-dimensional points; The coordinates of all the three-dimensional points are integrated to obtain training data. The obtained training data is then used to train the corresponding multilayer perceptrons to obtain two-dimensional coordinate prediction models for each semantic component.

[0010] Optionally, the three-dimensional model image processing method based on semantic segmentation and point sampling, wherein integrating all the three-dimensional point coordinates to obtain training data, and training the corresponding multilayer perceptrons with the obtained training data to obtain two-dimensional coordinate prediction models for each semantic component, specifically includes: The coordinates of all the three-dimensional points are integrated to obtain training data. The training data is then input into a multilayer perceptron for mapping to obtain multiple normalized two-dimensional coordinates. The initial two-dimensional coordinates are obtained by constraining all the normalized two-dimensional coordinates using the Sigmoid activation function of the multilayer perceptron. By inversely mapping all the initial two-dimensional coordinates, multiple target three-dimensional point coordinates are obtained; The multilayer perceptron performs multilayer nonlinear transformations on the coordinates of all the target three-dimensional points to obtain a two-dimensional coordinate prediction model.

[0011] Optionally, the 3D model image processing method based on semantic segmentation and point sampling, wherein obtaining the semantic components of the 3D mesh model to be unfolded, and predicting the coordinates of the semantic components using the 2D coordinate prediction model to obtain a position unfolding map, specifically includes: Obtain the semantic components of the three-dimensional mesh model to be unfolded, and traverse the semantic components to obtain multiple mesh vertices; The coordinates of all the grid vertices are predicted using the two-dimensional coordinate prediction model to obtain multiple coordinate values. All the coordinate values ​​are then integrated to obtain a position unfolded map.

[0012] Optionally, the 3D model image processing method based on semantic segmentation and point sampling, wherein the step of predicting the coordinates of all the grid vertices using the 2D coordinate prediction model to obtain multiple coordinate values, and integrating all the coordinate values ​​to obtain a position unfolded map, specifically includes: The coordinates of all the grid vertices are predicted by the two-dimensional coordinate prediction model to obtain multiple coordinate values. All the coordinate values ​​are then subjected to an affine transformation to obtain a UV coordinate set. Multiple target triangular facets are determined based on the UV coordinate set, and all the target triangular facets are connected to obtain a position unfolding diagram.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a three-dimensional model image processing system based on semantic segmentation and point sampling, wherein the three-dimensional model image processing system based on semantic segmentation and point sampling: The semantic segmentation module for the 3D mesh model is used to acquire the 3D mesh model to be unfolded, and to perform semantic segmentation processing on the 3D mesh model to be unfolded through a semantic segmentation network to obtain a semantic 3D mesh model. The two-dimensional coordinate model training module is used to obtain the surface area ratio, perform point sampling on the semantic three-dimensional mesh model according to the surface area ratio, obtain training data, and train the corresponding multilayer perceptrons with the obtained training data to obtain two-dimensional coordinate prediction models for each semantic component. The 3D model unfolding module is used to obtain the semantic components of the 3D mesh model to be unfolded, and to predict the coordinates of the semantic components through the 2D coordinate prediction model to obtain the position unfolding map.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a three-dimensional model image processing program based on semantic segmentation and point sampling, and the three-dimensional model image processing program based on semantic segmentation and point sampling, when executed by a processor, implements the steps of the three-dimensional model image processing method based on semantic segmentation and point sampling as described above.

[0015] This invention acquires a 3D mesh model to be unfolded, performs semantic segmentation on the model using a semantic segmentation network, and obtains a semantic 3D mesh model. It then acquires the surface area ratio, samples points from the semantic 3D mesh model based on this ratio to obtain training data, and trains corresponding multilayer perceptrons using this training data to obtain 2D coordinate prediction models for each semantic component. Finally, it acquires the semantic components of the 3D mesh model to be unfolded, predicts their coordinates using the 2D coordinate prediction models, and obtains a position unfolded map. This invention, based on semantic segmentation to divide components, trains multilayer perceptrons by sampling according to surface area ratios, predicts the position coordinates of each component, and generates a semantically corresponding position unfolded map, achieving accurate semantic understanding of the unfolded 3D model. Attached Figure Description

[0016] Figure 1 This is a flowchart of a preferred embodiment of the three-dimensional model image processing method based on semantic segmentation and point sampling of the present invention; Figure 2 This is a schematic diagram of the three-dimensional model unfolding process of a preferred embodiment of the three-dimensional model image processing method based on semantic segmentation and point sampling of the present invention; Figure 3This is a schematic diagram of the three-dimensional model unfolding process of a preferred embodiment of the three-dimensional model image processing method based on semantic segmentation and point sampling of the present invention; Figure 4 This is a structural diagram of a preferred embodiment of the three-dimensional model image processing system based on semantic segmentation and point sampling of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the device of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] Traditional 3D model UV unwrapping methods (such as manual segmentation and flattening, and neural network-based global mapping) are highly dependent on manual intervention and have limited universality. Furthermore, they lack an understanding of the semantic information at the model component level. This results in UV coordinate unwrapping maps that, while potentially having low overall distortion, lack clear semantic correspondence between UV coordinates. This hinders independent and refined texture editing and downstream processing of specific semantic components, making it difficult to achieve accurate semantic understanding of the 3D model unwrapping map. Therefore, a 3D model image processing method based on semantic segmentation and point sampling is needed. This method divides components based on semantic segmentation, trains a multilayer perceptron by sampling according to surface area ratios, predicts the position coordinates of each component, and generates a semantically corresponding position unwrapping map, thus achieving accurate semantic understanding of the 3D model unwrapping map.

[0019] The preferred embodiment of the present invention describes a 3D model image processing method based on semantic segmentation and point sampling, such as... Figure 1 and Figure 2 As shown, the 3D model image processing method based on semantic segmentation and point sampling includes the following steps: Step S10: Obtain the three-dimensional mesh model to be unfolded, and perform semantic segmentation processing on the three-dimensional mesh model to be unfolded through a semantic segmentation network to obtain a semantic three-dimensional mesh model.

[0020] Step S10 includes: Step S11: Obtain vertex coordinates, patch index, normal vector, and texture coordinates; normalize and unify the coordinates and scales of the vertex coordinates, patch index, normal vector, and texture coordinates to obtain preprocessed data. Step S12: Perform semantic segmentation on the preprocessed data through the encoder path and decoder path of the semantic segmentation network to obtain a semantic 3D mesh model.

[0021] Specifically, vertex coordinates, patch indices, normal vectors, and texture coordinates are obtained. The vertex coordinates, patch indices, normal vectors, and texture coordinates are then normalized and scaled (first, the model undergoes an integrity check, including verifying the validity of vertices and patches, removing duplicate vertices or non-manifold edges, and ensuring the correct mesh topology). This yields preprocessed data. The preprocessed data is then semantically segmented using the encoder and decoder paths of a semantic segmentation network to obtain a semantic 3D mesh model (the preprocessed 3D mesh model serves as the input for semantic segmentation, and the output is a normalized .obj file, where the original connection relationships of vertex and patch data are preserved, but the coordinate values ​​have been normalized). The 3D mesh model to be unfolded includes vertex coordinates, patch indices, normal vectors, and texture coordinates (the 3D mesh model file to be processed is read from local storage or network storage; the file format is .obj, which contains the model's vertex coordinates, patch indices, normal vectors, and texture coordinates).

[0022] As an example, the model undergoes preprocessing, including coordinate normalization and scale unification: the model is translated to the vicinity of the origin and scaled to within a unit cube, with all vertex coordinates normalized to [-1,1] or [0,1] to eliminate the impact of scale differences on subsequent processing. During preprocessing, open-source libraries (such as LiBIGL or OpenMesh) are used to implement mesh reading and basic operations, ensuring the model is in a standard 3D coordinate system.

[0023] Step S12 includes: Step S121: Normalize the preprocessed data according to the grid simplification algorithm to obtain a normalized grid; Step S122: Obtain the 3D mesh data set and cross-entropy loss. Through the encoder path and decoder path of the semantic segmentation network, perform semantic segmentation processing on the normalized mesh according to the 3D mesh data set and the cross-entropy loss to obtain a semantic 3D mesh model.

[0024] Specifically, the preprocessed data is normalized using a mesh simplification algorithm to obtain a normalized mesh (to ensure consistent network input size, the input mesh is first normalized to a predefined number of edges. This process uses a mesh simplification algorithm measured by geometric error to reduce the number of mesh edges to a target value while preserving the original shape features to the maximum extent. The normalized mesh serves as a fixed-size input for the network). A 3D mesh dataset and cross-entropy loss are then obtained (the mesh semantic segmentation network is trained under supervision using a 3D mesh dataset with point-level or polygon-level semantic annotations. The loss function uses cross-entropy loss to minimize the difference between the edge labels predicted by the network and the true labels). Through the encoder and decoder paths of the semantic segmentation network, the normalized mesh is semantically segmented based on the 3D mesh dataset and the cross-entropy loss to obtain a semantic 3D mesh model.

[0025] In this embodiment, the semantic segmentation network employs a U-shaped encoder-decoder architecture specifically designed for triangular meshes. The semantic segmentation network generally comprises an encoder path consisting of four downsampling stages, a bottleneck layer at the deepest level of the network, and a decoder path symmetrical to the encoder, consisting of four upsampling stages. The semantic segmentation network uses skip connections to concatenate the feature maps after each downsampling stage of the encoder with the input feature maps of the corresponding upsampling stages of the decoder, thereby combining the shallow geometric details captured by the encoder with the spatial structure recovered by the decoder, significantly improving the accuracy of part boundary segmentation. Each downsampling stage in the encoder path consists of two consecutive mesh convolutional layers and one mesh pooling layer, used to extract and compress mesh features layer by layer while expanding the receptive field. Symmetrically, each upsampling stage in the decoder path consists of a mesh unpooling layer and two consecutive mesh convolutional layers, responsible for recovering the spatial resolution of the feature maps layer by layer and refining semantic information. The bottleneck layer between the encoder and decoder consists of two consecutive mesh convolutional layers, used to capture and integrate the highest-level global semantic context information, without any pooling or upsampling operations.

[0026] As an example, the mesh convolutional layer performs convolution operations directly on the edges of the mesh, and its receptive field is explicitly defined as the target edge and its four neighboring edges (i.e., the remaining four edges in its two adjacent triangular facets). To resolve the inherent order ambiguity in the triangular mesh neighborhood, this layer divides the features of the four neighboring edges into two pairs according to their alignment in the two adjacent facets, and simultaneously calculates the absolute difference and sum for each pair of features, thereby generating a new neighborhood feature set with permutation invariance. Then, standard convolution weighted summation and nonlinear activation function transformation are performed. The mesh pooling layer is located at the end of each downsampling stage in the encoder path, and achieves feature map downsampling and mesh simplification through task-driven edge folding operations. This layer does not perform uniform pooling, but dynamically determines the priority of edge folding based on the L2 norm of the edge feature vectors learned by the network, thus prioritizing the folding of edges that contribute less to the current segmentation task, realizing an adaptive downsampling mechanism that simplifies the mesh while actively preserving the key regional structures related to semantic components.

[0027] Furthermore, the optimizer chosen is Adam, coupled with a learning rate decay strategy. After training, the network assigns a semantic component label to each edge of the input mesh. The final output is a .obj file with semantic information, which records the semantic ID of each basic unit (usually a face, whose label is determined by voting from the labels of its three edges) through vertex attributes, face attributes, or newly added edge attribute fields, thus obtaining a semantic 3D mesh model with clearly defined component semantic information.

[0028] Step S20: Obtain the surface area ratio, perform point sampling on the semantic 3D mesh model according to the surface area ratio to obtain training data, and train the corresponding multilayer perceptrons with the obtained training data to obtain a 2D coordinate prediction model for each semantic component.

[0029] Step S20 includes: Step S21: Obtain the surface area ratio. Based on the inverse transform sampling method, perform point sampling on the semantic 3D mesh model according to the surface area ratio to obtain triangular facets. Step S22: Select the triangular facets according to the centroid coordinate interpolation method to obtain multiple three-dimensional point coordinates; Step S23: Integrate all the three-dimensional point coordinates to obtain training data, and use the obtained training data to train the corresponding multilayer perceptrons to obtain two-dimensional coordinate prediction models for each semantic component.

[0030] Specifically, the surface area ratio and multilayer perceptron are obtained (first, the surface area of ​​each semantic component is calculated, and the number of sampling points is allocated proportionally according to its area in the total surface area of ​​the model, so as to ensure that each component can obtain a statistically representative set of points). Based on the inverse transformation sampling method, the semantic 3D mesh model is sampled according to the surface area ratio to obtain triangular patches (uniform point sampling is performed on the surface of each component, using the inverse transformation sampling method based on the triangle area, i.e., triangular patches are randomly selected according to the area ratio first). The triangular patches are selected according to the centroid coordinate interpolation method to obtain multiple 3D point coordinates (i.e., triangular patches are randomly selected according to the area ratio first, and then 3D point coordinates are randomly generated inside the selected triangles through centroid coordinate interpolation). All the 3D point coordinates are integrated to obtain training data (this process strictly limits the sampling points to be located within the surface area of ​​their respective semantic components, effectively avoiding fuzzy sampling at the component boundaries, and providing accurate and unbiased training data for subsequent independent training of UV coordinate prediction models for each component). The training data is input into the multilayer perceptron for training to obtain a 2D coordinate prediction model.

[0031] Step S23 includes: Step S231: Integrate all the three-dimensional point coordinates to obtain training data, and input the training data into a multilayer perceptron for mapping to obtain multiple normalized two-dimensional coordinates; Step S232: Constrain all the normalized two-dimensional coordinates using the Sigmoid activation function of the multilayer perceptron to obtain the initial two-dimensional coordinates; Step S233: Inversely map all the initial two-dimensional coordinates to obtain the coordinates of multiple target three-dimensional points; Step S234: Perform multi-layer nonlinear transformation on the coordinates of all the target three-dimensional points using the multilayer perceptron to obtain a two-dimensional coordinate prediction model.

[0032] Specifically, a two-dimensional coordinate prediction model is obtained by performing multi-layer nonlinear transformations on the coordinates of all target three-dimensional points using the multilayer perceptron. An initial two-dimensional coordinate is obtained by constraining all normalized two-dimensional coordinates using the sigmoid activation function of the multilayer perceptron (the sigmoid activation function strictly restricts the coordinates to the range [0, 1]). All the initial two-dimensional coordinates are then inversely mapped to obtain multiple target three-dimensional point coordinates (mapping the initial two-dimensional coordinates back to the corresponding three-dimensional point coordinates on the surface of the three-dimensional model). A two-dimensional coordinate prediction model is then obtained by performing multi-layer nonlinear transformations on the coordinates of all target three-dimensional points using the multilayer perceptron.

[0033] Furthermore, by independently training an MLP group for each semantic component and jointly optimizing the aforementioned loss function, and using the Adam optimizer for iterative training until the loss converges, a set of two-dimensional coordinate prediction models that can generate high-quality, low-distortion models for each semantic component while preserving the component's intrinsic geometric properties is finally obtained.

[0034] In this embodiment, each semantic component's MLP group contains two core networks: a texture coordinate MLP (Multilayer Perceptron), which is responsible for mapping its input 3D point coordinates to normalized 2D UV coordinates. The network consists of multiple fully connected layers, with nonlinear activation functions used between layers. The final output layer uses a sigmoid activation function to strictly limit the coordinates to the range [0,1], ensuring that the UV coordinates lie within a unit square. The surface coordinate MLP is the inverse mapping of the texture coordinate MLP, and its function is to map the 2D UV coordinates back to the corresponding 3D point coordinates on the surface of the 3D model. Its network structure is symmetrical to the texture coordinate MLP, achieving 2D-to-3D reconstruction through multiple nonlinear transformations.

[0035] As an example, the training of a multilayer perceptron employs a self-supervised paradigm, achieved by optimizing a comprehensive loss function. This loss function core includes the following terms: Cyclic consistency loss, which mandates that when a 3D point is mapped to UV space via texture coordinates MLP and then back to 3D space via surface coordinates MLP, its coordinates should be as consistent as possible with the origin; similarly, when a UV coordinate is mapped to 3D space via surface coordinates MLP and then back to UV space via texture coordinates MLP, its coordinates should also be as consistent as possible with the origin. This aims to ensure that forward and backward mappings are inverse processes, thereby guaranteeing bidirectional consistency and reversibility of the mapping. Surface consistency loss, by calculating the Chamfer distance, constrains the spatial distribution of the 3D point set reconstructed from random UV coordinates by surface coordinates MLP to be consistent with the original model surface sampling point set, ensuring that the UV space can completely cover the real model surface of the semantic component and avoiding mapping to invalid regions. Distortion loss, composed of conformal and uniform stretching terms, penalizes angular distortion and non-uniform scaling during UV mapping to promote low-distortion UV unwrapping results and ensure visual fidelity during texture mapping.

[0036] Step S30: Obtain the semantic components of the three-dimensional mesh model to be unfolded, and predict the coordinates of the semantic components using the two-dimensional coordinate prediction model to obtain the position unfolding map.

[0037] Step S30 includes: Step S31: Obtain the semantic components of the three-dimensional mesh model to be unfolded, and traverse the semantic components to obtain multiple mesh vertices; Step S32: Predict the coordinates of all the grid vertices using the two-dimensional coordinate prediction model to obtain multiple coordinate values. Integrate all the coordinate values ​​to obtain a position unfolded map.

[0038] Specifically, the semantic components of the three-dimensional mesh model to be unfolded are obtained, and the semantic components are traversed to obtain multiple mesh vertices (for each semantic component, all the mesh vertices contained therein are traversed, and the three-dimensional coordinates of the mesh vertices are input into the texture coordinate MLP dedicated to that component. The MLP directly outputs the coordinates of the vertex in the two-dimensional UV space based on the learned mapping relationship from 3D to 2D, and ensures that the output coordinate values ​​are all within the range of [0,1]). The coordinates of all the mesh vertices are predicted by the two-dimensional coordinate prediction model to obtain multiple coordinate values. All the coordinate values ​​are integrated to obtain the position unfolding map.

[0039] Step S32 includes: Step S321: Predict the coordinates of all the mesh vertices using the two-dimensional coordinate prediction model to obtain multiple coordinate values, and perform an affine transformation on all the coordinate values ​​to obtain a UV coordinate set. Step S322: Determine multiple target triangular facets based on the UV coordinate set, and connect all the target triangular facets to obtain a position unfolding diagram.

[0040] Specifically, the coordinates of all the mesh vertices are predicted using the two-dimensional coordinate prediction model to obtain multiple coordinate values. All these coordinate values ​​are then subjected to an affine transformation to obtain a UV coordinate set (with a non-overlapping square sub-region allocated within a unit UV space). Subsequently, the UV coordinates of the vertices within each component are subjected to an affine transformation, linearly scaling and translating them from the original [0, 1] range to the allocated sub-region to obtain the UV coordinate set. Multiple target triangular facets are determined based on the UV coordinate set, and all these target triangular facets are connected to obtain a position unfolding diagram (combined with the vertex-facet connection relationship of the three-dimensional mesh to generate a structured and complete position unfolding diagram).

[0041] like Figure 3As shown, the model preprocessing process regularizes the input 3D mesh model to obtain a normalized 3D model that meets the specific input format of the semantic segmentation model. The semantic segmentation process performs semantic segmentation on the normalized 3D model preprocessed by the above modules, assigning component labels to each edge of the model and outputting a mesh model with semantic information, which is a prerequisite for realizing semantic UV islands. The point sampling process uniformly samples the surface of the semantically information-rich mesh model according to the proportion of the mesh surface area, thus providing unbiased data covering the entire model surface for subsequent MLP training. The UV coordinate prediction process independently trains multiple MLPs in a supervised or self-supervised manner using the sampled point set. Each MLP specifically learns the 3D-to-2D mapping of a semantic component, and its network structure and loss function are specifically designed to reduce UV distortion. The UV unfolding process calls the pre-trained component MLPs in the UV coordinate prediction module to predict the UV coordinates of all its vertices, integrates and lays these coordinates into the same UV space, generating a complete UV unfolded map with semantic UV islands. The system output supports fine-grained texture editing based on component semantics.

[0042] Furthermore, such as Figure 4 As shown, based on the above-mentioned 3D model image processing method based on semantic segmentation and point sampling, the present invention also provides a 3D model image processing system based on semantic segmentation and point sampling, wherein the 3D model image processing system based on semantic segmentation and point sampling includes: The 3D mesh model semantic segmentation module 51 is used to acquire the 3D mesh model to be unfolded, and to perform semantic segmentation processing on the 3D mesh model to be unfolded through a semantic segmentation network to obtain a semantic 3D mesh model. The two-dimensional coordinate model training module 52 is used to obtain the surface area ratio, perform point sampling on the semantic three-dimensional mesh model according to the surface area ratio, obtain training data, and train the corresponding multilayer perceptrons with the obtained training data to obtain two-dimensional coordinate prediction models for each semantic component. The 3D model unfolding module 53 is used to obtain the semantic components of the 3D mesh model to be unfolded, and to predict the coordinates of the semantic components through the 2D coordinate prediction model to obtain the position unfolding map.

[0043] Furthermore, such as Figure 5 As shown, based on the above-mentioned three-dimensional model image processing method and system based on semantic segmentation and point sampling, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0044] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a 3D model image processing program 40 based on semantic segmentation and point sampling, which can be executed by the processor 10 to implement the 3D model image processing method based on semantic segmentation and point sampling described in this application.

[0045] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the three-dimensional model image processing method based on semantic segmentation and point sampling.

[0046] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminals communicate with each other via a system bus.

[0047] In one embodiment, when the processor 10 executes the 3D model image processing program 40 based on semantic segmentation and point sampling in the memory 20, the following steps are performed: A 3D mesh model to be unfolded is obtained, and a semantic segmentation network is used to perform semantic segmentation processing on the 3D mesh model to be unfolded to obtain a semantic 3D mesh model. Obtain the surface area ratio, perform point sampling on the semantic 3D mesh model according to the surface area ratio to obtain training data, and train the corresponding multilayer perceptrons with the obtained training data to obtain a 2D coordinate prediction model for each semantic component. Obtain the semantic components of the three-dimensional mesh model to be unfolded, and predict the coordinates of the semantic components using the two-dimensional coordinate prediction model to obtain the position unfolding map; The three-dimensional mesh model to be unfolded includes vertex coordinates, patch indices, normal vectors, and texture coordinates; The process of obtaining the 3D mesh model to be unfolded, and performing semantic segmentation on the 3D mesh model using a semantic segmentation network to obtain a semantic 3D mesh model, specifically includes: Obtain vertex coordinates, patch indexes, normal vectors, and texture coordinates; normalize and unify the coordinates and scales of the vertex coordinates, patch indexes, normal vectors, and texture coordinates to obtain preprocessed data. The preprocessed data is semantically segmented using the encoder and decoder paths of the semantic segmentation network to obtain a semantic 3D mesh model.

[0048] Specifically, the step of performing semantic segmentation processing on the preprocessed data using a semantic segmentation network to obtain a semantic 3D mesh model includes: The preprocessed data is normalized according to the grid simplification algorithm to obtain a normalized grid; A three-dimensional mesh data set and cross-entropy loss are obtained. The normalized mesh is then semantically segmented based on the three-dimensional mesh data set and the cross-entropy loss through the encoder and decoder paths of the semantic segmentation network to obtain a semantic three-dimensional mesh model.

[0049] The step of obtaining the surface area ratio, performing point sampling on the semantic 3D mesh model based on the surface area ratio to obtain training data, and training the corresponding multilayer perceptrons with the obtained training data to obtain a 2D coordinate prediction model for each semantic component specifically includes: The surface area ratio is obtained, and based on the inverse transform sampling method, point sampling is performed on the semantic 3D mesh model according to the surface area ratio to obtain triangular patches. The triangular facets are selected using the centroid coordinate interpolation method to obtain the coordinates of multiple three-dimensional points; The coordinates of all the three-dimensional points are integrated to obtain training data. The obtained training data is then used to train the corresponding multilayer perceptrons to obtain two-dimensional coordinate prediction models for each semantic component.

[0050] Specifically, the process of integrating all the three-dimensional point coordinates to obtain training data, and then using the obtained training data to train the corresponding multilayer perceptrons to obtain two-dimensional coordinate prediction models for each semantic component, includes: The coordinates of all the three-dimensional points are integrated to obtain training data. The training data is then input into a multilayer perceptron for mapping to obtain multiple normalized two-dimensional coordinates. The initial two-dimensional coordinates are obtained by constraining all the normalized two-dimensional coordinates using the Sigmoid activation function of the multilayer perceptron. By inversely mapping all the initial two-dimensional coordinates, multiple target three-dimensional point coordinates are obtained; The multilayer perceptron performs multilayer nonlinear transformations on the coordinates of all the target three-dimensional points to obtain a two-dimensional coordinate prediction model.

[0051] Specifically, obtaining the semantic components of the three-dimensional mesh model to be unfolded, and predicting the coordinates of the semantic components using the two-dimensional coordinate prediction model to obtain a position unfolding map, includes: Obtain the semantic components of the three-dimensional mesh model to be unfolded, and traverse the semantic components to obtain multiple mesh vertices; The coordinates of all the grid vertices are predicted using the two-dimensional coordinate prediction model to obtain multiple coordinate values. All the coordinate values ​​are then integrated to obtain a position unfolded map.

[0052] Specifically, the step of predicting the coordinates of all the grid vertices using the two-dimensional coordinate prediction model to obtain multiple coordinate values, and then integrating all the coordinate values ​​to obtain a position unfolded map, includes: The coordinates of all the grid vertices are predicted by the two-dimensional coordinate prediction model to obtain multiple coordinate values. All the coordinate values ​​are then subjected to an affine transformation to obtain a UV coordinate set. Multiple target triangular facets are determined based on the UV coordinate set, and all the target triangular facets are connected to obtain a position unfolding diagram.

[0053] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a three-dimensional model image processing program based on semantic segmentation and point sampling, and the three-dimensional model image processing program based on semantic segmentation and point sampling, when executed by a processor, implements the steps of the three-dimensional model image processing method based on semantic segmentation and point sampling as described above.

[0054] In summary, this invention provides a method, system, terminal, and storage medium for 3D model image processing based on semantic segmentation and point sampling. The method includes: acquiring a 3D mesh model to be unfolded; performing semantic segmentation processing on the 3D mesh model to be unfolded using a semantic segmentation network to obtain a semantic 3D mesh model; acquiring the surface area ratio; performing point sampling on the semantic 3D mesh model according to the surface area ratio to obtain training data; training corresponding multilayer perceptrons with the obtained training data to obtain two-dimensional coordinate prediction models for each semantic component; acquiring the semantic components of the 3D mesh model to be unfolded; and predicting the coordinates of the semantic components using the two-dimensional coordinate prediction models to obtain a position unfolded map. This invention divides components based on semantic segmentation, trains multilayer perceptrons by sampling according to surface area ratios, predicts the position coordinates of each component, and generates a semantically corresponding position unfolded map, achieving accurate semantic understanding of the unfolded 3D model map.

[0055] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal system that includes that element.

[0056] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0057] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A three-dimensional model image processing method based on semantic segmentation and point sampling, characterized in that, The 3D model image processing method based on semantic segmentation and point sampling includes: A 3D mesh model to be unfolded is obtained, and a semantic segmentation network is used to perform semantic segmentation processing on the 3D mesh model to be unfolded to obtain a semantic 3D mesh model. Obtain the surface area ratio, perform point sampling on the semantic 3D mesh model according to the surface area ratio to obtain training data, and train the corresponding multilayer perceptrons with the obtained training data to obtain a 2D coordinate prediction model for each semantic component. Obtain the semantic components of the three-dimensional mesh model to be unfolded, and predict the coordinates of the semantic components using the two-dimensional coordinate prediction model to obtain the position unfolding map.

2. The three-dimensional model image processing method based on semantic segmentation and point sampling according to claim 1, characterized in that, The three-dimensional mesh model to be unfolded includes vertex coordinates, patch indices, normal vectors, and texture coordinates; The process of obtaining the 3D mesh model to be unfolded, and performing semantic segmentation on the 3D mesh model using a semantic segmentation network to obtain a semantic 3D mesh model, specifically includes: Obtain vertex coordinates, patch indexes, normal vectors, and texture coordinates; normalize and unify the coordinates and scales of the vertex coordinates, patch indexes, normal vectors, and texture coordinates to obtain preprocessed data. The preprocessed data is semantically segmented using the encoder and decoder paths of the semantic segmentation network to obtain a semantic 3D mesh model.

3. The three-dimensional model image processing method based on semantic segmentation and point sampling according to claim 2, characterized in that, The step of performing semantic segmentation on the preprocessed data using a semantic segmentation network to obtain a semantic 3D mesh model specifically includes: The preprocessed data is normalized according to the grid simplification algorithm to obtain a normalized grid; A three-dimensional mesh data set and cross-entropy loss are obtained. The normalized mesh is then semantically segmented based on the three-dimensional mesh data set and the cross-entropy loss through the encoder and decoder paths of the semantic segmentation network to obtain a semantic three-dimensional mesh model.

4. The three-dimensional model image processing method based on semantic segmentation and point sampling according to claim 1, characterized in that, The process of obtaining the surface area ratio, sampling points in the semantic 3D mesh model based on the surface area ratio to obtain training data, and training the corresponding multilayer perceptrons with the obtained training data to obtain a 2D coordinate prediction model for each semantic component specifically includes: The surface area ratio is obtained, and based on the inverse transform sampling method, point sampling is performed on the semantic 3D mesh model according to the surface area ratio to obtain triangular patches. The triangular facets are selected using the centroid coordinate interpolation method to obtain the coordinates of multiple three-dimensional points; The coordinates of all the three-dimensional points are integrated to obtain training data. The obtained training data is then used to train the corresponding multilayer perceptrons to obtain two-dimensional coordinate prediction models for each semantic component.

5. The three-dimensional model image processing method based on semantic segmentation and point sampling according to claim 4, characterized in that, The process involves integrating all the three-dimensional point coordinates to obtain training data, and then using this training data to train corresponding multilayer perceptrons to obtain two-dimensional coordinate prediction models for each semantic component. Specifically, this includes: The coordinates of all the three-dimensional points are integrated to obtain training data. The training data is then input into a multilayer perceptron for mapping to obtain multiple normalized two-dimensional coordinates. The initial two-dimensional coordinates are obtained by constraining all the normalized two-dimensional coordinates using the Sigmoid activation function of the multilayer perceptron. By inversely mapping all the initial two-dimensional coordinates, multiple target three-dimensional point coordinates are obtained; The multilayer perceptron performs multilayer nonlinear transformations on the coordinates of all the target three-dimensional points to obtain a two-dimensional coordinate prediction model.

6. The three-dimensional model image processing method based on semantic segmentation and point sampling according to claim 5, characterized in that, The process of obtaining the semantic components of the three-dimensional mesh model to be unfolded, and predicting the coordinates of the semantic components using the two-dimensional coordinate prediction model to obtain a position unfolding map, specifically includes: Obtain the semantic components of the three-dimensional mesh model to be unfolded, and traverse the semantic components to obtain multiple mesh vertices; The coordinates of all the grid vertices are predicted using the two-dimensional coordinate prediction model to obtain multiple coordinate values. All the coordinate values ​​are then integrated to obtain a position unfolded map.

7. The three-dimensional model image processing method based on semantic segmentation and point sampling according to claim 6, characterized in that, The process of predicting the coordinates of all grid vertices using the two-dimensional coordinate prediction model to obtain multiple coordinate values, and then integrating all the coordinate values ​​to obtain a position unfolded map, specifically includes: The coordinates of all the grid vertices are predicted by the two-dimensional coordinate prediction model to obtain multiple coordinate values. All the coordinate values ​​are then subjected to an affine transformation to obtain a UV coordinate set. Multiple target triangular facets are determined based on the UV coordinate set, and all the target triangular facets are connected to obtain a position unfolding diagram.

8. A three-dimensional model image processing system based on semantic segmentation and point sampling, characterized in that, The 3D model image processing system based on semantic segmentation and point sampling includes: The semantic segmentation module for the 3D mesh model is used to acquire the 3D mesh model to be unfolded, and to perform semantic segmentation processing on the 3D mesh model to be unfolded through a semantic segmentation network to obtain a semantic 3D mesh model. The two-dimensional coordinate model training module is used to obtain the surface area ratio, perform point sampling on the semantic three-dimensional mesh model according to the surface area ratio, obtain training data, and train the corresponding multilayer perceptrons with the obtained training data to obtain two-dimensional coordinate prediction models for each semantic component. The 3D model unfolding module is used to obtain the semantic components of the 3D mesh model to be unfolded, and to predict the coordinates of the semantic components through the 2D coordinate prediction model to obtain the position unfolding map.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a 3D model image processing program based on semantic segmentation and point sampling stored in the memory and executable on the processor. When the 3D model image processing program based on semantic segmentation and point sampling is executed by the processor, it implements the steps of the 3D model image processing method based on semantic segmentation and point sampling as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a three-dimensional model image processing program based on semantic segmentation and point sampling, which, when executed by a processor, implements the steps of the three-dimensional model image processing method based on semantic segmentation and point sampling as described in any one of claims 1-7.

Images