A commodity image three-dimensional interaction anchor point automatic construction method and system
By automatically constructing 3D interactive anchor points for product images using pluggable 3D generative models and generative base models, the problem of large workload for manual annotation and anchor point drift is solved, achieving efficient and accurate 3D interactive anchor point generation.
Patent Information
- Application Number
- CN202611313594.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, after generating a 3D mesh from a product image, interactive anchor points need to be manually labeled, which is labor-intensive. Furthermore, overlay rendering causes frame-level latency and anchor point drift, resulting in inaccurate clicks.
A pluggable 3D generative model asynchronously generates 3D mesh assets. Combined with geometric semantic joint analysis and generative base model, the pose and semantic interpretation of interactive anchor points are automatically located. The anchor points are used as native slot sub-elements of the 3D viewer and transformed to the screen position frame by frame according to the projection matrix, controlling the hiding of the back anchor points.
It enables the automatic construction of interactive anchor points without manual annotation, eliminating anchor point drift and click inaccuracy, and improving the accuracy and efficiency of 3D interaction.
Smart Images

Figure CN123118992A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D visualization technology, and in particular to a method and system for automatically constructing 3D interactive anchor points for product images. Background Technology
[0002] In metaverse product display and e-commerce scenarios, merchants often only possess two-dimensional images of their products and wish to showcase them using rotatable and interactive three-dimensional digital twins, allowing buyers to click on product components to view information such as materials and parameters. In existing technologies, after generating a 3D mesh from the product image, the locations of interactive components must be manually marked on the 3D mesh, and explanations written, which is extremely labor-intensive when dealing with a large number of components. Furthermore, when rendering interactive anchor points using an overlay method, the overlay itself calculates the view and projection matrix, resulting in frame-level latency compared to the rendering within the 3D viewer, causing interactive anchor point drift and inaccurate clicks during rotation. Summary of the Invention
[0003] In view of this, the purpose of this invention is to propose a method and system for automatically constructing three-dimensional interactive anchor points for product images.
[0004] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows: In a first aspect, the present invention provides a method for automatically constructing three-dimensional interactive anchor points for product images, comprising: Acquire a product image, input the product image into a pluggable 3D generation model, and the pluggable 3D generation model asynchronously generates a 3D mesh asset corresponding to the product image. Geometric and semantic joint analysis is performed on the 3D mesh assets. Edges and seams are located by normal field mutation detection, and protrusions and depressions are located by curvature extremum analysis. The 3D mesh assets are segmented into components based on connectivity and normal clustering to obtain several semantic components. Multi-view rendering is performed on each semantic component. The results of the multi-view rendering and the geometric features of the semantic components are input into the generative base model. The generative base model outputs the semantic labels and interpretations of the semantic components, and determines the pose of the corresponding interaction anchor points based on the centroid and normal of the semantic components. Each interactive anchor point is treated as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to the screen position frame by frame according to its own projection matrix, and controls the hiding of the interactive anchor points behind the screen according to the data-driven visibility attribute, thus generating the final 3D interactive anchor points of the product.
[0005] In some embodiments, a product image is input to a pluggable 3D generation model, and the pluggable 3D generation model asynchronously generates a 3D mesh asset corresponding to the product image, including: Abrupt changes in the normal field of the 3D mesh asset are detected, and regions with drastic changes in normal field are identified as candidate regions for edges and seams. Extremum analysis is performed on the curvature field of the 3D mesh asset to locate the regions of maximum or minimum curvature as candidate regions for convexity and depression. Based on candidate regions for edges and seams, and candidate regions for protrusions and depressions, and combined with the connectivity constraints and normal clustering results of the 3D mesh assets, component-level segmentation is performed on the 3D mesh assets to obtain several semantic components. Each semantic component is rendered from multiple perspectives to generate a set of multi-view rendered images for each semantic component. The multi-view rendered image set is concatenated with the geometric feature vectors of each semantic component to form a multimodal feature representation; Multimodal feature representations are input into a generative base model, which outputs semantic labels and natural language interpretations for each semantic component. The pose of each interaction anchor point is determined by using the three-dimensional centroid coordinates of each semantic component as the spatial position of the interaction anchor point and the average normal of each semantic component as the orientation of the interaction anchor point.
[0006] In some embodiments, the multi-view rendered image set is concatenated with the geometric feature vectors of each semantic component to form a multimodal feature representation, including: Convolutional neural network encoding is performed on each view image in the multi-view rendering image set of each semantic component to extract the visual feature map of each view image; The geometric feature vectors of each semantic component are encoded using a fully connected network to extract the geometric feature embeddings of each semantic component; Visual feature maps and geometric features are embedded in the feature channel dimension and concatenated to form a multimodal joint feature tensor for each semantic component; Multimodal feature representations are input into a generative base model, which outputs semantic labels and natural language interpretations for each semantic component, including: The multimodal joint feature tensor is input into a pre-trained multimodal generative base model, which includes a cross-modal attention mechanism module. The cross-modal attention mechanism module performs cross-attention calculation on the visual feature map and the geometric feature embedding to generate context-aware features that integrate visual and geometric information; The autoregressive decoder of the multimodal generative basic model generates semantic labels and natural language interpretations for each semantic component word by word based on context-aware features.
[0007] In some embodiments, a joint geometric-semantic analysis is performed on the 3D mesh asset. Edges and seams are located through normal field mutation detection, protrusions and depressions are located through curvature extremum analysis, and the 3D mesh asset is segmented into components based on connectivity and normal clustering to obtain several semantic components, including: Abrupt changes are detected in the normal field of the 3D mesh asset. The normal angle between adjacent triangular facets is calculated. Edges with normal angles exceeding a preset abrupt change threshold are marked as edge candidate edges. The regions enclosed by edge candidate edges are marked as edge and seam candidate regions. Extremum analysis is performed on the curvature field of the 3D mesh asset to calculate the principal curvature value at each vertex. Vertices whose principal curvature values exceed a preset convexity threshold or fall below a preset concaveness threshold are marked as convexity and concaveness candidate vertices. Connected regions formed by convexity and concaveness candidate vertices are marked as convexity and concaveness candidate regions. Using edge and seam candidate regions and protrusion and depression candidate regions as seed regions, in the remaining regions of the 3D mesh asset, region growth and merging are performed based on the connectivity between triangular facets and the similarity of normal directions. Adjacent triangular facets with normal direction deviations within the preset normal tolerance range are grouped into the same candidate component to obtain several candidate components. Post-processing is performed on the candidate components to remove noisy components with an area smaller than a preset area threshold, and adjacent candidate components with similar normal directions are merged to obtain several semantic components.
[0008] In some embodiments, candidate regions for edges and seams, and candidate regions for protrusions and depressions are used as seed regions. In the remaining regions of the 3D mesh asset, region growth and merging are performed based on the connectivity between triangular faces and the similarity of their normal directions. Adjacent triangular faces with normal direction deviations within a preset normal tolerance range are grouped into the same candidate component, resulting in several candidate components, including: The triangular faces in the candidate regions of edges and seams and the candidate regions of convexities and depressions are marked as visited seed faces, and each visited seed face is added to the corresponding region growth queue as an initial seed point. Pop seed faces sequentially from the growth queues of each region, traverse the adjacent triangular faces of the seed faces, calculate the angle between the normal of the adjacent triangular face and the normal of the seed face, if the angle is less than the preset normal tolerance range and the adjacent triangular face has not been visited, mark the adjacent triangular face as visited and add it to the current region growth queue. Repeat the pop-up and traversal operations until the current region growth queue is empty. Then, all the triangular faces traversed in the current region growth queue are grouped into the same candidate component to obtain several candidate components. Post-processing is performed on candidate components to remove noisy components with an area smaller than a preset area threshold, and adjacent candidate components with similar normal directions are merged to obtain several semantic components, including: Calculate the surface area of each candidate component, and mark the candidate components with a surface area less than a preset area threshold as noise components and remove them from the candidate component set; For the remaining candidate components, calculate the average normal angle between adjacent candidate components. If the average normal angle is less than the preset normal merging threshold, then merge two adjacent candidate components into one semantic component to obtain several semantic components.
[0009] In some embodiments, multi-view rendering is performed on each semantic component, and the results of the multi-view rendering and the geometric features of the semantic components are input into a generative base model. The generative base model outputs the semantic labels and interpretations of the semantic components, and determines the pose of the corresponding interaction anchor points based on the centroid and normal of the semantic components, including: For each semantic component, multiple virtual camera poses are evenly distributed on the outer sphere of the semantic component. With the center of the bounding box of the semantic component as the viewpoint center, the semantic component is orthogonally projected and rendered from each virtual camera pose to generate a set of multi-view rendering images corresponding to each semantic component. Extract the geometric features of each semantic component, including vertex coordinate statistics, normal distribution histogram, curvature distribution histogram, and shape diameter function of each semantic component; The multi-view rendered image set is fused with geometric features at the feature level, and a multimodal joint feature representation is formed through feature concatenation operation; The multimodal joint feature representation is input into a pre-trained generative base model, which then outputs the semantic labels of each semantic component and the corresponding natural language interpretations based on the multimodal joint feature representation. Calculate the mean 3D coordinates of all vertices of each semantic component, and use the mean 3D coordinates as the spatial position of the corresponding interaction anchor point. Calculate the mean vector of the normals of all triangular facets of each semantic component, and use the mean vector as the orientation of the corresponding interaction anchor point to determine the pose of each interaction anchor point.
[0010] In some embodiments, for each semantic component, multiple virtual camera poses are uniformly arranged on the circumscribing sphere of the semantic component. Using the center of the bounding box of the semantic component as the viewpoint center, orthogonal projection rendering is performed on the semantic component from each virtual camera pose to generate a multi-view rendered image set corresponding to each semantic component, including: Calculate the axis-aligned bounding box of the semantic component, and use the center of the axis-aligned bounding box as the viewpoint center; Construct the circumspherical surface of the semantic component with the geometric center of the sphere as the center and a radius that is a fixed multiple of the length of the diagonal of the axis-aligned bounding box as the radius; A preset number of sampling points are set up on the outer sphere according to the Fibonacci spherical uniform sampling algorithm, and each sampling point is used as the position coordinate of the virtual camera. The direction from the position coordinates of each virtual camera to the center of the viewpoint is taken as the observation direction of each virtual camera. The projection method of each virtual camera is set to orthogonal projection, and the view volume of each virtual camera is set to the outer cube of the axis-aligned bounding box. The semantic components are rendered from the poses of each virtual camera, generating a set of multi-view rendered images corresponding to each semantic component; The multi-view rendered image set is fused with geometric features at the feature level, and a multimodal joint feature representation is formed through feature concatenation operations, including: Convolutional neural network encoding is performed on each view image in the multi-view rendering image set of each semantic component to extract the visual feature map of each view image; The geometric features of each semantic component are encoded using a fully connected network, and the geometric feature embeddings of each semantic component are extracted. Visual feature maps and geometric features are embedded in the feature channel dimension and spliced together to form a multimodal joint feature tensor for each semantic component.
[0011] In some embodiments, each interactive anchor point is treated as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to its screen position frame by frame according to its own projection matrix, and manages the hiding of the back interactive anchor points according to data-driven visibility attributes, generating the final 3D interactive anchor points for the product, including: Register each interactive anchor point as a child node in the native slot of the 3D viewer scene graph, and set the spatial position coordinates and orientation vector of each interactive anchor point in the local coordinate system of the 3D mesh asset. In each rendering frame of the 3D viewer, the current view matrix and projection matrix of the 3D viewer are obtained. Based on the view matrix and projection matrix, the spatial position coordinates and orientation vectors of each interactive anchor point are transformed from the local coordinate system to the world coordinate system, camera coordinate system and clipping coordinate system in turn to obtain the two-dimensional coordinates and depth values of each interactive anchor point in the screen space. Based on the comparison between the depth value of each interactive anchor point in screen space and the depth value of the 3D mesh asset at the corresponding pixel position, a visibility attribute is set for each interactive anchor point. When the depth value of the interactive anchor point is greater than the depth value of the 3D mesh asset, it is determined that the interactive anchor point is behind the screen and the visibility attribute is marked as hidden. Based on the visibility attribute of each interactive anchor point, only interactive anchor points with the visibility attribute set to "display" are rendered in each rendering frame, and each interactive anchor point is output synchronously with the screen rendered by the 3D viewer to generate the final 3D interactive anchor points for the product.
[0012] In some embodiments, the method further includes: The 3D mesh assets are added to the asset library and their rendering parameters are configured. The first frame of the 3D mesh assets is then captured as a thumbnail. When each interactive anchor point is used as a native slot sub-element in the 3D viewer, the thumbnail is bound to the corresponding interactive anchor point as a visual identifier of the interactive anchor point in the 3D viewer. While the 3D viewer renders each interactive anchor point, thumbnails bound to each interactive anchor point are displayed synchronously, allowing users to preview the image of the corresponding component before clicking on the interactive anchor point.
[0013] In a second aspect, the present invention also provides an automatic construction system for three-dimensional interactive anchor points of product images, used to perform the method described in the first aspect, the system comprising: Multi-source image-generated 3D units are used to acquire product images and input the product images into a pluggable 3D generation model, which then asynchronously generates 3D mesh assets corresponding to the product images. The geometric semantic anchor inference unit is used to perform joint geometric and semantic analysis on 3D mesh assets. It locates edges and seams through normal field mutation detection, locates protrusions and depressions through curvature extremum analysis, and segments the 3D mesh assets into several semantic components based on connectivity and normal clustering. It performs multi-view rendering on each semantic component, and inputs the results of the multi-view rendering and the geometric features of the semantic components into the generative base model. The generative base model outputs the semantic labels and interpretations of the semantic components, and determines the pose of the corresponding interactive anchor points based on the centroid and normal of the semantic components. The anchor point rendering unit is used to treat each interactive anchor point as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to the screen position frame by frame according to its own projection matrix, and controls the hiding of the interactive anchor points behind the screen according to the data-driven visibility attribute, thus generating the final 3D interactive anchor points of the product.
[0014] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: This invention provides an automatic construction method for 3D interactive anchor points of product images. The method involves acquiring a product image, inputting it into a pluggable 3D generation model, which asynchronously generates a 3D mesh asset corresponding to the product image. Geometric and semantic joint analysis is performed on the 3D mesh asset. Edges and seams are located through normal field mutation detection, and protrusions and depressions are located through curvature extremum analysis. The 3D mesh asset is then segmented into several semantic components based on connectivity and normal clustering. Multi-view rendering is performed on each semantic component, and the results of the multi-view rendering and the geometric features of the semantic components are input into a generative base model. The generative base model outputs semantic labels and interpretations for the semantic components, and determines the pose of the corresponding interactive anchor points based on the centroid and normal of the semantic components. Each interactive anchor point is used as a native slot sub-element of a 3D viewer. The 3D viewer transforms each interactive anchor point to its screen position frame by frame based on its own projection matrix, and manages the hiding of back-side interactive anchor points based on data-driven visibility attributes, generating the final 3D interactive anchor points for the product. This invention automatically infers the pose and semantic meaning of interactive anchor points through geometric-semantic joint analysis, eliminating the need for manual annotation. At the same time, the interactive anchor points are used as native slot sub-elements of the 3D viewer, and the projection matrix of the 3D viewer transforms them frame by frame, so that the interactive anchor points are in the same frame as the screen, thus eliminating drift and click inaccuracy during rotation. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic diagram of the overall process of the method described in the specific implementation method; Figure 2 This is a schematic diagram illustrating the asynchronous generation of 3D mesh assets from product images and automatic storage of the pluggable 3D generation model as described in the specific implementation method. Figure 3 This is a schematic diagram illustrating the geometric and semantic joint analysis of three-dimensional mesh assets as described in the specific implementation method; Figure 4 This is a schematic diagram illustrating the inference of the pose and semantic interpretation of the interactive anchor point as described in the specific implementation method; Figure 5 This is a schematic diagram illustrating how the interactive anchor point, as a native slot sub-element of the 3D viewer, is transformed frame by frame by its own projection matrix in the specific implementation method. Figure 6 This is a schematic diagram of the overall process of the system described in the specific implementation method.
[0017] Figure label: 100. Multi-source image-generated 3D units; 200. Asset Receiving Unit; 300. Geometric semantic anchor point inference unit; 400, Anchor Point Rendering Unit. Detailed Implementation
[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] Please see Figures 1 to 5 In a first aspect, this embodiment provides a method for automatically constructing three-dimensional interactive anchor points for product images, including: Acquire a product image, input the product image into a pluggable 3D generation model, and the pluggable 3D generation model asynchronously generates a 3D mesh asset corresponding to the product image. Geometric and semantic joint analysis is performed on the 3D mesh assets. Edges and seams are located by normal field mutation detection, and protrusions and depressions are located by curvature extremum analysis. The 3D mesh assets are segmented into components based on connectivity and normal clustering to obtain several semantic components. Multi-view rendering is performed on each semantic component. The results of the multi-view rendering and the geometric features of the semantic components are input into the generative base model. The generative base model outputs the semantic labels and interpretations of the semantic components, and determines the pose of the corresponding interaction anchor points based on the centroid and normal of the semantic components. Each interactive anchor point is treated as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to the screen position frame by frame according to its own projection matrix, and controls the hiding of the interactive anchor points behind the screen according to the data-driven visibility attribute, thus generating the final 3D interactive anchor points of the product.
[0020] In this embodiment, the product image is input into the pluggable 3D generation model, which infers the 3D geometric structure of the product object based on the visual features in the product image and generates the corresponding 3D mesh asset asynchronously, thereby avoiding blocking subsequent processing.
[0021] In this embodiment, curvature extremum analysis is used to locate regions where the curvature value reaches a local maximum or minimum. These regions correspond to the protrusions or depressions on the surface of the 3D mesh asset. Based on connectivity and normal clustering, triangular facets with similar normal directions and spatial connectivity are grouped into the same group, thereby dividing the 3D mesh asset into several semantic components. Each semantic component corresponds to an interactive functional area of the product object.
[0022] In this embodiment, the image data obtained from multi-view rendering and the geometric features of the semantic component are jointly input into the generative base model. The generative base model infers the semantic label and natural language interpretation of the semantic component based on the joint information of the visual and geometric modalities. The spatial location of the interaction anchor point is determined by using the spatial distribution center of all vertices of the semantic component as the spatial location of the interaction anchor point, and the overall orientation of the semantic component surface as the orientation of the interaction anchor point.
[0023] In this embodiment, the 3D viewer acquires its own view matrix and projection matrix in each rendering frame. Through matrix transformation, it progressively transforms each interactive anchor point from the local coordinate system to the screen coordinate system, ensuring that the interactive anchor points and the image rendered by the 3D viewer are in the same frame. Based on the depth comparison results between the interactive anchor points and the 3D mesh asset, a visibility attribute is set for each interactive anchor point. Only interactive anchor points facing the user are rendered, thereby controlling the hiding of interactive anchor points behind them, and ultimately generating the final 3D interactive anchor points for the product.
[0024] This embodiment asynchronously generates 3D mesh assets using a pluggable 3D generative model, avoiding blocking subsequent processes; it automatically segments semantic components through geometric-semantic joint analysis, providing a semantic basis for the positioning of interactive anchor points; it automatically infers the pose and semantic meaning of interactive anchor points by combining multi-view rendering with a generative base model, eliminating the need for manual annotation; and it eliminates anchor point drift and click inaccuracies caused by overlay rendering by registering interactive anchor points as native slot sub-elements and transforming them frame by frame according to the 3D viewer's own projection matrix, thereby achieving automatic construction and accurate rendering of 3D interactive anchor points for products without manual intervention.
[0025] Please see Figure 6 In a second aspect, this embodiment also provides an automatic construction system for three-dimensional interactive anchor points of product images, used to execute the method described in the first aspect, the system comprising: The multi-source image-generated 3D unit 100 is used to acquire product images and input the product images into the pluggable 3D generation model, which then asynchronously generates 3D mesh assets corresponding to the product images. The geometric semantic anchor point inference unit 300 is used to perform joint geometric and semantic analysis on 3D mesh assets. It locates edges and seams through normal field mutation detection, locates protrusions and depressions through curvature extremum analysis, and segments the 3D mesh assets into several semantic components based on connectivity and normal clustering. It performs multi-view rendering on each semantic component, inputs the results of multi-view rendering and the geometric features of the semantic components into the generative base model, outputs the semantic labels and interpretations of the semantic components, and determines the pose of the corresponding interactive anchor points based on the centroid and normal of the semantic components. Anchor point rendering unit 400 is used to treat each interactive anchor point as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to the screen position frame by frame according to its own projection matrix, and controls the hiding of the interactive anchor points behind the screen according to the data-driven visibility attribute, so as to generate the final 3D interactive anchor points of the product.
[0026] In this embodiment, the multi-source image-generated 3D unit 100 is used to acquire product images and input them into a pluggable 3D generation model. The pluggable 3D generation model asynchronously generates 3D mesh assets corresponding to the product images. The geometric semantic anchor point inference unit 300 is used to perform geometric semantic joint analysis on the 3D mesh assets. It locates edges and seams through normal field mutation detection, locates protrusions and depressions through curvature extremum analysis, and performs component segmentation on the 3D mesh assets based on connectivity and normal clustering to obtain several semantic components. It performs multi-view rendering on each semantic component, inputs the results of multi-view rendering and the geometric features of the semantic components into the generative base model, and outputs the semantic labels and interpretations of the semantic components. It determines the pose of the corresponding interactive anchor points based on the centroid and normal of the semantic components. The anchor point rendering unit 400 is used to use each interactive anchor point as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to the screen position frame by frame according to its own projection matrix, and controls the hiding of the back interactive anchor points according to data-driven visibility attributes to generate the final 3D interactive anchor points of the product.
[0027] This system embodiment and the method embodiment of the first aspect are based on the same technical concept, and they maintain a high degree of consistency in the identification of technical problems, the adoption of technical means, and the realization of technical effects. In this system embodiment, the multi-source image-generated 3D unit 100 corresponds to the step of acquiring a product image and asynchronously generating a 3D mesh asset in the method embodiment of the first aspect; the geometric semantic anchor point inference unit 300 corresponds to the step of performing geometric semantic joint analysis, multi-view rendering, and generative basic model inference in the method embodiment of the first aspect; and the anchor point rendering unit 400 corresponds to the step of rendering and controlling the visibility of interactive anchor points as native slot sub-elements in the method embodiment of the first aspect.
[0028] This embodiment implements the functions of each step in the first aspect of the method embodiment in a modular manner. The data flow and cooperation relationship between the units are consistent with the execution order between the steps in the method embodiment. The two correspond to and support each other in the overall solution.
[0029] In some embodiments, the method further includes: The 3D mesh assets are added to the asset library and their rendering parameters are configured. The first frame of the 3D mesh assets is then captured as a thumbnail. When each interactive anchor point is used as a native slot sub-element in the 3D viewer, the thumbnail is bound to the corresponding interactive anchor point as a visual identifier of the interactive anchor point in the 3D viewer. While the 3D viewer renders each interactive anchor point, thumbnails bound to each interactive anchor point are displayed synchronously, allowing users to preview the image of the corresponding component before clicking on the interactive anchor point.
[0030] In this embodiment, the first frame of the rendered 3D mesh asset is captured as a thumbnail, and the obtained thumbnail is used to quickly identify the corresponding 3D mesh asset in the interactive interface.
[0031] In this embodiment, the established binding relationship makes the thumbnail a visual identifier of the interactive anchor point in the 3D viewer, and users can quickly identify the appearance of the component corresponding to the interactive anchor point through the thumbnail.
[0032] In this embodiment, through a synchronous display mechanism, users can preview the image of the corresponding component before clicking the interactive anchor point, thereby improving the intuitiveness of the interactive preview.
[0033] This embodiment extracts the first frame of the rendered 3D mesh asset as a thumbnail and binds it to the corresponding interactive anchor point. The thumbnail is displayed synchronously when the 3D viewer renders the interactive anchor point, allowing users to preview the image of the corresponding component before clicking the interactive anchor point, thereby improving the interactive preview experience of the 3D interactive anchor point of the product.
[0034] Correspondingly, the system described in the second aspect may also include: The asset storage unit 200 is used to include the 3D mesh asset into the asset library and configure rendering parameters for it, and to capture the first frame of the rendering of the 3D mesh asset as a thumbnail.
[0035] In this embodiment, the asset storage unit 200 is used to include the 3D mesh asset into the asset library and configure its rendering parameters, and to capture the first frame of the 3D mesh asset rendering as a thumbnail.
[0036] The asset storage unit 200 in this system embodiment belongs to the same system architecture as the multi-source image-generated 3D unit 100, geometric semantic anchor point inference unit 300, and anchor point rendering unit 400 disclosed in the aforementioned second aspect system embodiment. The asset storage unit 200 intervenes after the multi-source image-generated 3D unit 100 generates 3D mesh assets, performing storage and parameter configuration operations on the generated 3D mesh assets. This provides the geometric semantic anchor point inference unit 300 with standardized 3D mesh assets that have a consistent visual presentation for subsequent geometric semantic joint analysis. The thumbnails captured by the asset storage unit 200 can be further transmitted to the anchor point rendering unit 400 as visual identifiers for interactive anchor points in the 3D viewer.
[0037] With the addition of the asset storage unit 200, the system described in the second aspect forms a complete closed loop in the entire chain of 3D mesh asset generation, storage, analysis, and rendering, and the data flow and collaboration between the units are closer.
[0038] In some embodiments, a product image is input to a pluggable 3D generation model, and the pluggable 3D generation model asynchronously generates a 3D mesh asset corresponding to the product image, including: Abrupt changes in the normal field of the 3D mesh asset are detected, and regions with drastic changes in normal field are identified as candidate regions for edges and seams. Extremum analysis is performed on the curvature field of the 3D mesh asset to locate the regions of maximum or minimum curvature as candidate regions for convexity and depression. Based on candidate regions for edges and seams, and candidate regions for protrusions and depressions, and combined with the connectivity constraints and normal clustering results of the 3D mesh assets, component-level segmentation is performed on the 3D mesh assets to obtain several semantic components. Each semantic component is rendered from multiple perspectives to generate a set of multi-view rendered images for each semantic component. The multi-view rendered image set is concatenated with the geometric feature vectors of each semantic component to form a multimodal feature representation; Multimodal feature representations are input into a generative base model, which outputs semantic labels and natural language interpretations for each semantic component. The pose of each interaction anchor point is determined by using the three-dimensional centroid coordinates of each semantic component as the spatial position of the interaction anchor point and the average normal of each semantic component as the orientation of the interaction anchor point.
[0039] In this embodiment, the area to be located is the location of the edge or seam on the surface of the 3D mesh asset, and such areas are marked as edge and seam candidate areas.
[0040] In this embodiment, the located area is the location of the protrusions or depressions on the surface of the three-dimensional mesh asset, and such areas are marked as candidate areas for protrusions and depressions.
[0041] In this embodiment, based on candidate regions for edges and seams, candidate regions for protrusions and depressions, and combined with connectivity constraints between triangular faces in the 3D mesh asset and normal clustering results, component-level segmentation is performed on the 3D mesh asset. By grouping triangular faces with similar normal directions and spatial connectivity into the same group, several semantic components are obtained, each semantic component corresponding to an interactive functional area of the product object.
[0042] In this embodiment, the generated multi-view rendered image set of each semantic component is used to provide a visual representation of the semantic component from different perspectives.
[0043] In this embodiment, the spliced multimodal feature representation integrates information from the visual modality and the geometric modality, and is used to provide joint input for subsequent semantic inference.
[0044] In this embodiment, the multimodal feature representation is input into the generative base model. Based on the combined visual and geometric information in the multimodal feature representation, the generative base model outputs semantic labels and natural language interpretations for each semantic component. Semantic labels are used to identify the category of the semantic component, and natural language interpretations are used to describe the function or attribute of the semantic component.
[0045] In this embodiment, the three-dimensional centroid coordinates are determined by the average coordinates of all vertices of the semantic component, and the average normal is determined by the average vector of the normals of all triangular facets of the semantic component, thereby determining the pose of each interactive anchor point.
[0046] This embodiment locates candidate regions through normal field mutation detection and curvature extremum analysis, performs component-level segmentation by combining connectivity constraints and normal clustering to obtain semantic components, forms multimodal feature representations through multi-view rendering and geometric feature splicing, inputs them to the generative base model to output semantic labels and interpretations, and determines the pose of interactive anchor points with centroid coordinates and average normal, thereby achieving automatic inference of interactive anchor points without human intervention.
[0047] In some embodiments, the multi-view rendered image set is concatenated with the geometric feature vectors of each semantic component to form a multimodal feature representation, including: Convolutional neural network encoding is performed on each view image in the multi-view rendering image set of each semantic component to extract the visual feature map of each view image; The geometric feature vectors of each semantic component are encoded using a fully connected network to extract the geometric feature embeddings of each semantic component; Visual feature maps and geometric features are embedded in the feature channel dimension and concatenated to form a multimodal joint feature tensor for each semantic component; Multimodal feature representations are input into a generative base model, which outputs semantic labels and natural language interpretations for each semantic component, including: The multimodal joint feature tensor is input into a pre-trained multimodal generative base model, which includes a cross-modal attention mechanism module. The cross-modal attention mechanism module performs cross-attention calculation on the visual feature map and the geometric feature embedding to generate context-aware features that integrate visual and geometric information; The autoregressive decoder of the multimodal generative basic model generates semantic labels and natural language interpretations for each semantic component word by word based on context-aware features.
[0048] In this embodiment, the convolutional neural network extracts spatial structure information from the view image through layer-by-layer convolution and pooling operations, and the output visual feature map is used to characterize the visual response intensity of the view image at different spatial locations.
[0049] In this embodiment, the fully connected network maps geometric feature vectors to the embedding space through linear transformation and nonlinear activation. The output geometric feature embedding is used to characterize the distribution of the geometric attributes of semantic components in the embedding space.
[0050] In this embodiment, the splicing operation merges the feature channels of the visual modality and the feature channels of the geometric modality into a single tensor. The resulting multimodal joint feature tensor of each semantic component contains both visual and geometric information.
[0051] In this embodiment, the multimodal generative basic model includes a cross-modal attention mechanism module. The cross-modal attention mechanism module uses the visual feature map as the query and the geometric feature embedding as the key-value pair. Through attention calculation, each position in the visual feature map pays attention to the relevant position in the geometric feature embedding, thereby generating context-aware features that integrate visual and geometric information.
[0052] In this embodiment, the autoregressive decoder predicts the next word based on the generated word sequence and context-aware features at each generation step, thereby gradually constructing complete semantic tags and natural language interpretations.
[0053] This embodiment uses a convolutional neural network to encode visual feature maps and a fully connected network to encode geometric feature embeddings. These are then concatenated along the feature channel dimension to form a multimodal joint feature tensor. A cross-modal attention mechanism module is used to fuse visual and geometric information to generate context-aware features. An autoregressive decoder generates semantic labels and natural language interpretations word by word, thereby enabling automatic inference of semantic information of semantic components without human intervention.
[0054] In some embodiments, a joint geometric-semantic analysis is performed on the 3D mesh asset. Edges and seams are located through normal field mutation detection, protrusions and depressions are located through curvature extremum analysis, and the 3D mesh asset is segmented into components based on connectivity and normal clustering to obtain several semantic components, including: Abrupt changes are detected in the normal field of the 3D mesh asset. The normal angle between adjacent triangular facets is calculated. Edges with normal angles exceeding a preset abrupt change threshold are marked as edge candidate edges. The regions enclosed by edge candidate edges are marked as edge and seam candidate regions. Extremum analysis is performed on the curvature field of the 3D mesh asset to calculate the principal curvature value at each vertex. Vertices whose principal curvature values exceed a preset convexity threshold or fall below a preset concaveness threshold are marked as convexity and concaveness candidate vertices. Connected regions formed by convexity and concaveness candidate vertices are marked as convexity and concaveness candidate regions. Using edge and seam candidate regions and protrusion and depression candidate regions as seed regions, in the remaining regions of the 3D mesh asset, region growth and merging are performed based on the connectivity between triangular facets and the similarity of normal directions. Adjacent triangular facets with normal direction deviations within the preset normal tolerance range are grouped into the same candidate component to obtain several candidate components. Post-processing is performed on the candidate components to remove noisy components with an area smaller than a preset area threshold, and adjacent candidate components with similar normal directions are merged to obtain several semantic components.
[0055] In this embodiment, edges whose normal angle exceeds a preset abrupt change threshold are marked as edge candidate edges. These edges correspond to locations on the surface of the 3D mesh asset where the normal direction undergoes a sharp change. The regions enclosed by the edge candidate edges are marked as edge and seam candidate regions. These regions correspond to the locations of edges or seams on the surface of the 3D mesh asset.
[0056] In this embodiment, vertices with principal curvature values exceeding a preset convexity threshold are marked as convexity candidate vertices, and vertices with principal curvature values below a preset concavity threshold are marked as concavity candidate vertices. Connected regions formed by convexity candidate vertices or concavity candidate vertices are marked as convexity and concavity candidate regions, which correspond to the locations of convexities or concavities on the surface of the 3D mesh asset.
[0057] In this embodiment, candidate regions for edges and seams, and candidate regions for protrusions and depressions are used as seed regions. In the remaining regions of the 3D mesh asset, region growth and merging are performed based on the connectivity between triangular facets and the similarity of their normal directions. During the region growth process, adjacent triangular facets with normal direction deviations within a preset normal tolerance range are grouped into the same candidate component, thereby gradually expanding the coverage of candidate components to obtain several candidate components.
[0058] In this embodiment, the surface area of each candidate component is calculated, and candidate components with a surface area less than a preset area threshold are identified as noise components and removed from the candidate component set. For the remaining candidate components, the average normal angle between adjacent candidate components is calculated, and adjacent candidate components with an average normal angle less than a preset normal merging threshold are merged into a semantic component, thereby obtaining the final number of semantic components.
[0059] This embodiment locates candidate regions for edges and seams by calculating the included angle of normals and comparing it with a threshold, and locates candidate regions for protrusions and depressions by calculating the principal curvature value and comparing it with a threshold. Candidate regions are obtained by growing regions using the candidate regions as seeds. Semantic components are then obtained by post-processing through area filtering and normal merging. This achieves automatic component segmentation of 3D mesh assets without human intervention.
[0060] In some embodiments, candidate regions for edges and seams, and candidate regions for protrusions and depressions are used as seed regions. In the remaining regions of the 3D mesh asset, region growth and merging are performed based on the connectivity between triangular faces and the similarity of their normal directions. Adjacent triangular faces with normal direction deviations within a preset normal tolerance range are grouped into the same candidate component, resulting in several candidate components, including: The triangular faces in the candidate regions of edges and seams and the candidate regions of convexities and depressions are marked as visited seed faces, and each visited seed face is added to the corresponding region growth queue as an initial seed point. Pop seed faces sequentially from the growth queues of each region, traverse the adjacent triangular faces of the seed faces, calculate the angle between the normal of the adjacent triangular face and the normal of the seed face, if the angle is less than the preset normal tolerance range and the adjacent triangular face has not been visited, mark the adjacent triangular face as visited and add it to the current region growth queue. Repeat the pop-up and traversal operations until the current region growth queue is empty. Then, all the triangular faces traversed in the current region growth queue are grouped into the same candidate component to obtain several candidate components. Post-processing is performed on candidate components to remove noisy components with an area smaller than a preset area threshold, and adjacent candidate components with similar normal directions are merged to obtain several semantic components, including: Calculate the surface area of each candidate component, and mark the candidate components with a surface area less than a preset area threshold as noise components and remove them from the candidate component set; For the remaining candidate components, calculate the average normal angle between adjacent candidate components. If the average normal angle is less than the preset normal merging threshold, then merge two adjacent candidate components into one semantic component to obtain several semantic components.
[0061] In this embodiment, the marked visited seed patches are used as the starting points for region growth and are added to their respective region growth queues to ensure that the candidate components to which each seed patch belongs can grow independently.
[0062] In this embodiment, the angle between the normal of the adjacent triangular facet and the normal of the seed facet is calculated, and the size of the angle measures the degree of deviation between the two triangular facets in the normal direction. If the angle is less than a preset normal tolerance range and the adjacent triangular facet has not been visited, the adjacent triangular facet is marked as visited and added to the current region growth queue, thereby gradually expanding the coverage of the current candidate component.
[0063] In this embodiment, when the current region growth queue is empty, it means that the current candidate component can no longer grow. All triangular faces traversed by the current region growth queue are grouped into the same candidate component, resulting in several candidate components.
[0064] In this embodiment, the surface area of each candidate component is calculated, and candidate components with a surface area less than a preset area threshold are marked as noise components. Noise components are usually generated by random errors in normal or curvature detection, and they are removed from the candidate component set to filter out interference areas that do not meet the area requirements.
[0065] In this embodiment, the average normal angle is determined by the angle between the vector averages of the normals of all triangular facets in adjacent candidate components. If the average normal angle is less than a preset normal merging threshold, then two adjacent candidate components are merged into one semantic component, thereby integrating adjacent components with similar normal directions into a complete semantic component.
[0066] This embodiment drives region growth through seed patch marking and region growth queue, gradually expands candidate parts by using the normal angle as the growth criterion, removes noisy parts by area screening, and merges adjacent parts by comparing the normal angle, thereby achieving accurate part segmentation of 3D mesh assets without human intervention.
[0067] In some embodiments, multi-view rendering is performed on each semantic component, and the results of the multi-view rendering and the geometric features of the semantic components are input into a generative base model. The generative base model outputs the semantic labels and interpretations of the semantic components, and determines the pose of the corresponding interaction anchor points based on the centroid and normal of the semantic components, including: For each semantic component, multiple virtual camera poses are evenly distributed on the outer sphere of the semantic component. With the center of the bounding box of the semantic component as the viewpoint center, the semantic component is orthogonally projected and rendered from each virtual camera pose to generate a set of multi-view rendering images corresponding to each semantic component. Extract the geometric features of each semantic component, including vertex coordinate statistics, normal distribution histogram, curvature distribution histogram, and shape diameter function of each semantic component; The multi-view rendered image set is fused with geometric features at the feature level, and a multimodal joint feature representation is formed through feature concatenation operation; The multimodal joint feature representation is input into a pre-trained generative base model, which then outputs the semantic labels of each semantic component and the corresponding natural language interpretations based on the multimodal joint feature representation. Calculate the mean 3D coordinates of all vertices of each semantic component, and use the mean 3D coordinates as the spatial position of the corresponding interaction anchor point. Calculate the mean vector of the normals of all triangular facets of each semantic component, and use the mean vector as the orientation of the corresponding interaction anchor point to determine the pose of each interaction anchor point.
[0068] In this embodiment, the circumscribed sphere is constructed with the geometric center of the semantic component as its center. The uniform distribution of virtual camera poses on the circumscribed sphere ensures that visual information of the semantic component is captured from all directions. Using the center of the bounding box of the semantic component as the viewpoint center, orthogonal projection rendering is performed on the semantic component from each virtual camera pose. Orthogonal projection avoids the influence of perspective distortion on visual information. The resulting set of multi-view rendered images corresponding to each semantic component is used to provide distortion-free visual representations of the semantic component from different viewpoints.
[0069] In this embodiment, vertex coordinate statistics are used to characterize the spatial distribution range and dispersion of semantic components, the normal distribution histogram is used to characterize the statistical distribution of the normal directions of the semantic component's surface, the curvature distribution histogram is used to characterize the statistical distribution of the curvature of the semantic component's surface, and the shape diameter function is used to characterize the distance distribution from each point inside the semantic component to the surface. The extracted geometric features describe the geometric properties of the semantic component from multiple dimensions.
[0070] In this embodiment, through feature concatenation operation, the visual features obtained by encoding the multi-view rendered image set and the geometric features obtained by encoding the geometric features are merged in the feature dimension. The multimodal joint feature representation simultaneously includes the visual appearance information and geometric structure information of the semantic components.
[0071] In this embodiment, semantic tags are used to identify the category of semantic components, and natural language interpretations are used to describe the function or attributes of semantic components.
[0072] In this embodiment, the mean three-dimensional coordinates of all vertices in each semantic component are calculated. The mean three-dimensional coordinates are obtained by averaging the coordinates of all vertices in the semantic component across the three dimensions, and are used as the spatial position of the corresponding interaction anchor point. The mean vector of the normals of all triangular facets in each semantic component is calculated. This mean vector is obtained by summing and normalizing the normal vectors of all triangular facets in the semantic component, and is used as the orientation of the corresponding interaction anchor point. The pose of each interaction anchor point is determined by its spatial position and orientation.
[0073] This embodiment obtains a set of multi-view rendered images by uniformly arranging virtual camera poses on an circumscribed sphere and orthogonally projecting them. It combines geometric features such as vertex coordinate statistics, normal distribution histogram, curvature distribution histogram, and shape diameter function, and forms a multimodal joint feature representation through feature-level fusion. This representation is then input into a generative basic model to output semantic labels and interpretations. The spatial position and orientation of the interactive anchor point are determined by the mean of vertex coordinates and the mean of normal vectors. This allows for the automatic inference of the pose and semantic information of the interactive anchor point without human intervention.
[0074] In some embodiments, for each semantic component, multiple virtual camera poses are uniformly arranged on the circumscribing sphere of the semantic component. Using the center of the bounding box of the semantic component as the viewpoint center, orthogonal projection rendering is performed on the semantic component from each virtual camera pose to generate a multi-view rendered image set corresponding to each semantic component, including: Calculate the axis-aligned bounding box of the semantic component, and use the center of the axis-aligned bounding box as the viewpoint center; Construct the circumspherical surface of the semantic component with the geometric center of the sphere as the center and a radius that is a fixed multiple of the length of the diagonal of the axis-aligned bounding box as the radius; A preset number of sampling points are set up on the outer sphere according to the Fibonacci spherical uniform sampling algorithm, and each sampling point is used as the position coordinate of the virtual camera. The direction from the position coordinates of each virtual camera to the center of the viewpoint is taken as the observation direction of each virtual camera. The projection method of each virtual camera is set to orthogonal projection, and the view volume of each virtual camera is set to the outer cube of the axis-aligned bounding box. The semantic components are rendered from the poses of each virtual camera, generating a set of multi-view rendered images corresponding to each semantic component; The multi-view rendered image set is fused with geometric features at the feature level, and a multimodal joint feature representation is formed through feature concatenation operations, including: Convolutional neural network encoding is performed on each view image in the multi-view rendering image set of each semantic component to extract the visual feature map of each view image; The geometric features of each semantic component are encoded using a fully connected network, and the geometric feature embeddings of each semantic component are extracted. Visual feature maps and geometric features are embedded in the feature channel dimension and spliced together to form a multimodal joint feature tensor for each semantic component.
[0075] In this embodiment, the axis-aligned bounding box of the semantic component is calculated. The boundary of the axis-aligned bounding box is parallel to the coordinate axis direction, and the center of the axis-aligned bounding box is used as the viewpoint center. The viewpoint center is used to determine the target point in the subsequent virtual camera observation direction.
[0076] In this embodiment, the outer sphere completely encloses the semantic component, ensuring that the semantic component can be fully observed from any position on the sphere.
[0077] In this embodiment, the Fibonacci spherical uniform sampling algorithm uses the mapping relationship between the golden angle and spherical coordinates to ensure that sampling points are evenly distributed on the circumscribed sphere, avoiding sampling point clustering or omissions. Each sampling point is used as the position coordinate of the virtual camera.
[0078] In this embodiment, each virtual camera is ensured to face the central region of the semantic component, and the projection method of each virtual camera is set to orthographic projection, which eliminates the influence of perspective distortion on the rendering result. The view volume of each virtual camera is set to the outer cube of the axis-aligned bounding box, ensuring that the semantic component falls completely within the view volume.
[0079] In this embodiment, the generated multi-view rendered image set corresponding to each semantic component contains visual information captured from multiple uniformly distributed perspectives.
[0080] In this embodiment, the convolutional neural network extracts spatial structural features from the view image through layer-by-layer convolution operations, and the output visual feature map is used to characterize the visual response intensity of the view image at different spatial locations.
[0081] In this embodiment, the fully connected network maps geometric features to the embedding space through linear transformation, and the output geometric feature embedding is used to characterize the distribution of the geometric attributes of semantic components in the embedding space.
[0082] In this embodiment, the splicing operation merges the feature channels of the visual modality and the feature channels of the geometric modality into a single tensor. The resulting multimodal joint feature tensor of each semantic component contains both visual and geometric information.
[0083] In this embodiment, the viewpoint center and sampling range are determined by an axis-aligned bounding box and an outer sphere. The virtual camera pose is set up using a Fibonacci spherical uniform sampling algorithm. The rendering quality is ensured by orthogonal projection and view volume settings. Then, the visual feature map is encoded by a convolutional neural network and the geometric feature embedding is encoded by a fully connected network. The feature channel dimensions are spliced to form a multimodal joint feature tensor, thereby providing a joint input that integrates visual and geometric information for subsequent semantic inference.
[0084] In some embodiments, each interactive anchor point is treated as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to its screen position frame by frame according to its own projection matrix, and manages the hiding of the back interactive anchor points according to data-driven visibility attributes, generating the final 3D interactive anchor points for the product, including: Register each interactive anchor point as a child node in the native slot of the 3D viewer scene graph, and set the spatial position coordinates and orientation vector of each interactive anchor point in the local coordinate system of the 3D mesh asset. In each rendering frame of the 3D viewer, the current view matrix and projection matrix of the 3D viewer are obtained. Based on the view matrix and projection matrix, the spatial position coordinates and orientation vectors of each interactive anchor point are transformed from the local coordinate system to the world coordinate system, camera coordinate system and clipping coordinate system in turn to obtain the two-dimensional coordinates and depth values of each interactive anchor point in the screen space. Based on the comparison between the depth value of each interactive anchor point in screen space and the depth value of the 3D mesh asset at the corresponding pixel position, a visibility attribute is set for each interactive anchor point. When the depth value of the interactive anchor point is greater than the depth value of the 3D mesh asset, it is determined that the interactive anchor point is behind the screen and the visibility attribute is marked as hidden. Based on the visibility attribute of each interactive anchor point, only interactive anchor points with the visibility attribute set to "display" are rendered in each rendering frame, and each interactive anchor point is output synchronously with the screen rendered by the 3D viewer to generate the final 3D interactive anchor points for the product.
[0085] In this embodiment, each interactive anchor point is registered as a child node in the native slot of the 3D viewer scene graph. The native slot of the scene graph provides the interactive anchor point with a mounting location directly associated with the 3D viewer rendering pipeline. The registration operation makes the interactive anchor point part of the 3D viewer scene graph. Each interactive anchor point is assigned its spatial position coordinates and orientation vector in the local coordinate system of the 3D mesh asset. The set spatial position coordinates and orientation vector are used to determine the pose of the interactive anchor point in 3D space.
[0086] In this embodiment, the view matrix is used to transform coordinates from the world coordinate system to the camera coordinate system, and the projection matrix is used to transform coordinates from the camera coordinate system to the clipping coordinate system. Based on the view matrix and the projection matrix, the spatial position coordinates and orientation vectors of each interactive anchor point are sequentially transformed from the local coordinate system to the world coordinate system, the camera coordinate system, and the clipping coordinate system. After this step-by-step transformation, the two-dimensional coordinates and depth values of each interactive anchor point in screen space are obtained.
[0087] In this embodiment, the depth value is used to measure the distance from the object to the virtual camera. When the depth value of the interaction anchor point is greater than the depth value of the 3D mesh asset, it indicates that the interaction anchor point is located behind the 3D mesh asset. The interaction anchor point is determined to be behind the object, and the visibility attribute is marked as hidden.
[0088] In this embodiment, by filtering the visibility attribute, the interaction anchors on the back are prevented from obscuring the interaction anchors on the front or interfering with user operations.
[0089] This embodiment registers interactive anchor points as native slot child nodes in the scene graph, directly associating them with the rendering pipeline. Through step-by-step transformation of the view matrix and projection matrix, interactive anchor points are mapped from the local coordinate system to the screen space. The back interactive anchor points are determined by comparing depth values and their visibility attributes are set. The visibility attributes are used to filter and render only the front interactive anchor points and output them synchronously with the screen, thereby eliminating anchor point drift and misdisplay of back anchor points caused by overlay rendering.
[0090] This invention provides an automatic construction method for 3D interactive anchor points of product images. The method involves acquiring a product image, inputting it into a pluggable 3D generation model, which asynchronously generates a 3D mesh asset corresponding to the product image. Geometric and semantic joint analysis is performed on the 3D mesh asset. Edges and seams are located through normal field mutation detection, and protrusions and depressions are located through curvature extremum analysis. The 3D mesh asset is then segmented into several semantic components based on connectivity and normal clustering. Multi-view rendering is performed on each semantic component, and the results of the multi-view rendering and the geometric features of the semantic components are input into a generative base model. The generative base model outputs semantic labels and interpretations for the semantic components, and determines the pose of the corresponding interactive anchor points based on the centroid and normal of the semantic components. Each interactive anchor point is used as a native slot sub-element of a 3D viewer. The 3D viewer transforms each interactive anchor point to its screen position frame by frame based on its own projection matrix, and manages the hiding of back-side interactive anchor points based on data-driven visibility attributes, generating the final 3D interactive anchor points for the product. This invention automatically infers the pose and semantic meaning of interactive anchor points through geometric-semantic joint analysis, eliminating the need for manual annotation. At the same time, the interactive anchor points are used as native slot sub-elements of the 3D viewer, and the projection matrix of the 3D viewer transforms them frame by frame, so that the interactive anchor points are in the same frame as the screen, thus eliminating drift and click inaccuracy during rotation.
[0091] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0093] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A method for automatically constructing three-dimensional interactive anchor points of a product image, characterized in that, include: A product image is acquired and input into a pluggable 3D generation model, which then asynchronously generates a 3D mesh asset corresponding to the product image. Geometric and semantic joint analysis is performed on the three-dimensional mesh asset. Edges and seams are located by normal field mutation detection, and protrusions and depressions are located by curvature extremum analysis. The three-dimensional mesh asset is segmented into components based on connectivity and normal clustering to obtain several semantic components. Multi-view rendering is performed on each of the semantic components. The results of the multi-view rendering and the geometric features of the semantic components are input into the generative base model. The generative base model outputs the semantic labels and interpretations of the semantic components, and the pose of the corresponding interaction anchor points is determined by the centroid and normal of the semantic components. Each interactive anchor point is used as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to the screen position frame by frame according to its own projection matrix, and controls the hiding of the interactive anchor points behind the screen according to the data-driven visibility attribute, thus generating the final 3D interactive anchor points of the product.
2. The method of claim 1, wherein the method further comprises: The product image is input into a pluggable 3D generation model, which asynchronously generates a 3D mesh asset corresponding to the product image, including: Abrupt change detection is performed on the normal field of the three-dimensional mesh asset, and regions with drastic normal changes are identified as candidate regions for edges and seams; Extremum analysis is performed on the curvature field of the three-dimensional mesh asset to locate the regions of maximum or minimum curvature as candidate regions for convexity and depression. Based on the candidate regions of edges and seams, the candidate regions of protrusions and depressions, and combined with the connectivity constraints and normal clustering results of the 3D mesh asset, component-level segmentation is performed on the 3D mesh asset to obtain several semantic components. Each semantic component is rendered from multiple perspectives to generate a set of multi-view rendered images for each semantic component. The multi-view rendered image set is concatenated with the geometric feature vectors of each semantic component to form a multimodal feature representation; The multimodal feature representations are input into the generative base model, and the generative base model outputs the semantic labels and natural language interpretations of each semantic component. The pose of each interaction anchor point is determined by using the three-dimensional centroid coordinates of each semantic component as the spatial position of the interaction anchor point and the average normal of each semantic component as the orientation of the interaction anchor point.
3. The method of claim 2, wherein the method further comprises: The multi-view rendered image set is concatenated with the geometric feature vectors of each semantic component to form a multimodal feature representation, including: Each view image in the multi-view rendering image set of each semantic component is encoded by a convolutional neural network to extract the visual feature map of each view image; The geometric feature vectors of each semantic component are encoded using a fully connected network to extract the geometric feature embeddings of each semantic component; The visual feature map and the geometric feature are embedded in the feature channel dimension and concatenated to form a multimodal joint feature tensor for each semantic component; The multimodal feature representations are input into a generative base model, which outputs semantic labels and natural language interpretations for each semantic component, including: The multimodal joint feature tensor is input into a pre-trained multimodal generative base model, which includes a cross-modal attention mechanism module. The cross-modal attention mechanism module performs cross-attention calculation on the visual feature map and the geometric feature embedding to generate context-aware features that integrate visual and geometric information; The autoregressive decoder of the multimodal generative basic model generates semantic labels and natural language interpretations for each semantic component word by word based on the context-aware features.
4. The method of claim 1, wherein the method further comprises: A joint geometric-semantic analysis is performed on the 3D mesh asset. Edges and seams are located through normal field mutation detection, and protrusions and depressions are located through curvature extremum analysis. The 3D mesh asset is then segmented into components based on connectivity and normal clustering to obtain several semantic components, including: Abrupt detection is performed on the normal field of the three-dimensional mesh asset, the normal angle between adjacent triangular facets is calculated, the edge with the normal angle exceeding the preset abrupt change threshold is marked as an edge candidate edge, and the area enclosed by the edge candidate edge is marked as an edge and seam candidate area; Extreme value analysis is performed on the curvature field of the three-dimensional mesh asset to calculate the principal curvature value at each vertex. Vertices whose principal curvature values exceed a preset convexity threshold or fall below a preset concaveness threshold are marked as convexity and concaveness candidate vertices. The connected regions formed by the convexity and concaveness candidate vertices are marked as convexity and concaveness candidate regions. Using the candidate regions for edges and seams and the candidate regions for protrusions and depressions as seed regions, in the remaining regions of the three-dimensional mesh asset, region growth and merging are performed based on the connectivity between triangular facets and the similarity of normal directions. Adjacent triangular facets with normal direction deviations within a preset normal tolerance range are grouped into the same candidate component to obtain several candidate components. The candidate components are post-processed to remove noisy components with an area smaller than a preset area threshold, and adjacent candidate components with similar normal directions are merged to obtain several semantic components.
5. The method of claim 4, wherein the method further comprises: Using the candidate regions for edges and seams and the candidate regions for protrusions and depressions as seed regions, region growth and merging are performed in the remaining regions of the 3D mesh asset based on the connectivity between triangular facets and the similarity of their normal directions. Adjacent triangular facets with normal direction deviations within a preset normal tolerance range are grouped into the same candidate component, resulting in several candidate components, including: The triangular facets in the edge and seam candidate regions and the protrusion and depression candidate regions are marked as visited seed facets, and each visited seed facet is added as an initial seed point to the corresponding region growth queue. Seed faces are popped out sequentially from each region growth queue. The adjacent triangular faces of the seed face are traversed. The angle between the normal of the adjacent triangular face and the normal of the seed face is calculated. If the angle is less than the preset normal tolerance range and the adjacent triangular face has not been visited, the adjacent triangular face is marked as visited and added to the current region growth queue. Repeat the pop-up and traversal operations until the current region growth queue is empty. Then, all the triangular faces traversed in the current region growth queue are grouped into the same candidate component to obtain several candidate components. The candidate components are post-processed to remove noisy components with an area smaller than a preset area threshold, and adjacent candidate components with similar normal directions are merged to obtain several semantic components, including: Calculate the surface area of each candidate component, and mark candidate components with a surface area less than the preset area threshold as noise components and remove them from the candidate component set; For the remaining candidate components, calculate the average normal angle between adjacent candidate components. If the average normal angle is less than a preset normal merging threshold, then merge two adjacent candidate components into one semantic component to obtain several semantic components.
6. The method of claim 1, wherein the method further comprises: The semantic components are rendered using multi-view rendering. The results of the multi-view rendering and the geometric features of the semantic components are input into a generative base model. The generative base model outputs the semantic labels and definitions of the semantic components. The pose of the corresponding interaction anchor points is determined using the centroid and normal of the semantic components, including: For each semantic component, multiple virtual camera poses are evenly distributed on the outer sphere of the semantic component. With the center of the bounding box of the semantic component as the viewpoint center, the semantic component is orthogonally projected and rendered from each of the virtual camera poses to generate a set of multi-view rendered images corresponding to each semantic component. Extract the geometric features of each semantic component, including vertex coordinate statistics, normal distribution histogram, curvature distribution histogram, and shape diameter function of each semantic component; The multi-view rendered image set is fused with the geometric features at the feature level, and a multimodal joint feature representation is formed through feature concatenation operation; The multimodal joint feature representation is input into a pre-trained generative base model, which outputs semantic labels and corresponding natural language interpretations for each semantic component based on the multimodal joint feature representation. Calculate the mean three-dimensional coordinates of all vertices of each semantic component, and use the mean three-dimensional coordinates as the spatial position of the corresponding interaction anchor point. Calculate the mean vector of the normals of all triangular facets of each semantic component, and use the mean vector as the orientation of the corresponding interaction anchor point to determine the pose of each interaction anchor point.
7. The method of claim 6, wherein the method further comprises: determining a plurality of anchor points of the product image; and determining a plurality of anchor points of the three-dimensional model. For each semantic component, multiple virtual camera poses are uniformly arranged on the circumscribing surface of the semantic component. Using the center of the bounding box of the semantic component as the viewpoint center, orthogonal projection rendering is performed on the semantic component from each of the virtual camera poses to generate a multi-view rendering image set corresponding to each semantic component, including: Calculate the axis-aligned bounding box of the semantic component, and use the center of the axis-aligned bounding box as the viewpoint center; With the geometric center of the semantic component as the center of the sphere, and a fixed multiple of the diagonal length of the axis-aligned bounding box as the radius, construct the circumspherical surface of the semantic component; A preset number of sampling points are arranged on the outer sphere according to the Fibonacci spherical uniform sampling algorithm, and each sampling point is used as the position coordinate of the virtual camera. The direction from the position coordinates of each virtual camera to the center of the viewpoint is taken as the observation direction of each virtual camera. The projection mode of each virtual camera is set to orthogonal projection, and the view volume of each virtual camera is set to the outer cube of the axis-aligned bounding box. The semantic components are rendered from the poses of each virtual camera to generate a set of multi-view rendered images corresponding to each semantic component; The multi-view rendered image set is fused with the geometric features at the feature level, and a multimodal joint feature representation is formed through feature concatenation operations, including: Each view image in the multi-view rendering image set of each semantic component is encoded by a convolutional neural network to extract the visual feature map of each view image; The geometric features of each semantic component are encoded using a fully connected network to extract the geometric feature embeddings of each semantic component; The visual feature map and the geometric feature are embedded in the feature channel dimension and concatenated to form a multimodal joint feature tensor for each semantic component.
8. The method of claim 1, wherein the method further comprises: determining a plurality of anchor points of the product image; and determining a plurality of anchor points of the three-dimensional model. Each of the aforementioned interactive anchor points is used as a native slot sub-element of the 3D viewer. The 3D viewer transforms each of the interactive anchor points to the screen position frame by frame according to its own projection matrix, and manages the hiding of the back interactive anchor points according to data-driven visibility attributes, generating the final 3D interactive anchor points for the product, including: Register each of the interactive anchor points as a child node in the native slot of the 3D viewer scene graph, and set the spatial position coordinates and orientation vector of each interactive anchor point in the local coordinate system of the 3D mesh asset. In each rendering frame of the 3D viewer, the current view matrix and projection matrix of the 3D viewer are obtained. Based on the view matrix and the projection matrix, the spatial position coordinates and orientation vectors of each interactive anchor point are transformed from the local coordinate system to the world coordinate system, the camera coordinate system and the clipping coordinate system in sequence to obtain the two-dimensional coordinates and depth values of each interactive anchor point in the screen space. Based on the comparison between the depth value of each interactive anchor point in screen space and the depth value of the 3D mesh asset at the corresponding pixel position, a visibility attribute is set for each interactive anchor point. When the depth value of the interactive anchor point is greater than the depth value of the 3D mesh asset, it is determined that the interactive anchor point is located behind the screen, and the visibility attribute is marked as hidden. Based on the visibility attribute of each interactive anchor point, only interactive anchor points with the visibility attribute set to "display" are rendered in each rendering frame, and each interactive anchor point is output synchronously with the screen rendered by the 3D viewer to generate the final 3D interactive anchor points for the product.
9. The method for automatically constructing three-dimensional interactive anchor points for product images according to claim 1, characterized in that, The method further includes: The 3D mesh asset is added to the asset library and its rendering parameters are configured. The first frame of the rendering of the 3D mesh asset is captured as a thumbnail. When each of the interactive anchor points is used as a native slot sub-element of the 3D viewer, the thumbnail is bound to the corresponding interactive anchor point as a visual identifier of the interactive anchor point in the 3D viewer. When the 3D viewer renders each of the interactive anchor points, the thumbnail bound to each of the interactive anchor points is displayed synchronously, so that the user can preview the image of the corresponding component before clicking on the interactive anchor point.
10. A system for automatically constructing 3D interactive anchor points for product images, characterized in that, The system for performing the method according to any one of claims 1 to 9, the system comprising: A multi-source image-generated 3D unit is used to acquire product images and input the product images into a pluggable 3D generation model, which then asynchronously generates 3D mesh assets corresponding to the product images. The geometric semantic anchor point inference unit is used to perform joint geometric semantic analysis on the 3D mesh asset. It locates edges and seams through normal field mutation detection, locates protrusions and depressions through curvature extremum analysis, and divides the 3D mesh asset into several semantic components based on connectivity and normal clustering. It performs multi-view rendering on each semantic component, inputs the results of the multi-view rendering and the geometric features of the semantic components into the generative base model, outputs the semantic labels and interpretations of the semantic components, and determines the pose of the corresponding interactive anchor point based on the centroid and normal of the semantic component. An anchor point rendering unit is used to treat each interactive anchor point as a native slot sub-element of the 3D viewer. The 3D viewer transforms each interactive anchor point to the screen position frame by frame according to its own projection matrix, and controls the hiding of the interactive anchor points behind the screen according to the data-driven visibility attribute, so as to generate the final 3D interactive anchor points of the product.
