Printed matter mockup design method and system incorporating augmented reality technology

Abstract

This invention relates to the field of computer-aided design technology, and discloses a method and system for replicating printed materials using augmented reality technology. The method includes: reconstructing a dynamic scene encompassing geometric, material, and environmental dimensions through multimodal perception of the physical scene and using physically based rendering material inverse calculation technology; constructing a dynamic parametric design space to achieve real-time binding of virtual elements and reconstructed data, as well as dynamic topology adaptive adjustment; generating a closed loop based on dynamic parameters, and outputting intelligent design units conforming to CAD standards. The system consists of modules for multimodal perception, dynamic scene reconstruction, dynamic parametric design, and closed-loop generation and output. This invention improves design fidelity in non-rigid form carriers and produces digital assets that can be directly used in intelligent manufacturing.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided design, specifically relating to a method and system for designing printed reproductions that integrates augmented reality technology. Background Technology

[0002] In the fields of computer-aided design and digital manufacturing, the digital reproduction and re-creation of physical printed materials is a core link in the cultural and creative, packaging engineering and art design industries. It involves a series of complex engineering processes such as geometric reconstruction, color restoration and graphic information superposition of existing printed media.

[0003] Augmented reality technology, as an interactive medium connecting physical and digital spaces, plays an increasingly important role in the design verification and interactive presentation of printed materials. By aligning and overlaying virtual design elements with real printing scenes in spatial dimensions, it can provide designers with intuitive spatial perception and visual reference, assisting in the digital mapping and secondary design of physical entities.

[0004] Existing technologies still face limitations when dealing with the reproduction of printed materials under non-ideal conditions. This is mainly because augmented reality technology is positioned as a mere visualization tool, detached from the core of creation. This leads to an over-reliance on manual image registration and feature extraction in the design process. When dealing with non-rigid, non-planar carriers such as bent paper, wrinkles, or irregularly shaped packaging, it is difficult to eliminate geometric errors and color distortions caused by perspective distortion, uneven lighting, and material substrate deformation. Consequently, the dynamic matching degree between virtual elements and physical surfaces is insufficient, and it is impossible to extract material parameters and ambient lighting information with physical rendering attributes. This serious disconnect between design logic and physical reality leads to a structural contradiction between efficiency and accuracy, making it difficult to generate structured design models with high fidelity, physical consistency, and manufacturability. Therefore, developing a reproduction design solution that can deeply integrate augmented reality technology and dynamic parametric modeling has become a technical challenge that the industry urgently needs to solve. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for printing reproduction design that integrates augmented reality technology, which can effectively solve the problems in the background technology mentioned above, namely, the structural contradiction between reproduction design efficiency and accuracy caused by the existing technology using augmented reality technology only as a visualization tool, and the technical pain point of lack of robustness in non-ideal physical environments.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a method for designing and replicating printed materials that integrates augmented reality technology, comprising the following steps: Step S1: Capture the physical scene of the target printed material in real time through a multimodal sensing device, and perform dynamic scene reconstruction based on physical rendering material inverse calculation technology to generate reconstruction data including geometric dimension, material dimension and environmental dimension. Step S2: Construct an interactive, dynamically parameterized design space based on the reconstructed data, and bind the virtual design elements to the reconstructed data in real time and dynamically adjust the topology adaptively; Step S3: Generate a closed loop based on the dynamic parameterized data during the design interaction process, encapsulate and output intelligent design units that conform to computer-aided design standards.

[0007] Preferably, step S1 specifically includes the following steps: Step S11: Collect surface information of the target printed matter and its carrier using depth sensing equipment and spectral sensing equipment; Step S12: Extract a high-precision geometric model of the target printed surface using a point cloud reconstruction algorithm to record the non-ideal geometric features of the carrier; Step S13: By analyzing the interaction between incident and reflected light, the basic color, roughness, and metallicity parameters of the printed surface are calculated in reverse. Step S14: Estimate the ambient light occlusion information and light source direction information in the physical scene in real time to complete the extraction of global illumination information.

[0008] Preferably, step S2 specifically includes the following steps: Step S21: Establish parametric design anchor points in the augmented reality view and identify the designer's interaction commands; Step S22: Generate design elements in the virtual space based on the interactive instructions, and map the design elements to the geometric model using a dynamic topology adaptive algorithm; Step S23: Perform physically based visual rendering on the mapped design elements according to the material dimension parameters and global illumination information.

[0009] Preferably, the non-ideal geometric features recorded in step S12 include the bending deformation of the paper, surface wrinkles, and the irregular topological structure of the physical carrier. During the point cloud reconstruction process, the system uses a sampling strategy with a preset step size to scan the physical surface and locates geometrically abrupt change regions through curvature analysis to construct a dynamic mesh model that can reflect the deformation state of the physical entity in real time.

[0010] Preferably, the reverse calculation process of the basic color, roughness and metallicity parameters mentioned in step S13 is as follows: establish a reflectance distribution function model based on micro-surface theory, take the radiance data obtained by the spectral sensing device as input, solve the surface reflectance equation through iterative optimization algorithm, extract the intrinsic physical properties of the printed surface, and eliminate the interference of ambient light source on the color characterization of the material itself.

[0011] Preferably, the ambient light occlusion information extraction process in step S14 includes: performing spherical harmonic function projection based on the reconstructed geometric model, calculating the proportion of each point in space occluded by the surrounding geometric structure, and generating an ambient light occlusion layer. The extraction of light source direction information is achieved by analyzing the distribution characteristics of the highlight areas on the physical surface and inversely calculating the vector coordinates of the main light source in three-dimensional space.

[0012] Preferably, the dynamic topology adaptive algorithm in step S22 is as follows: when establishing a mapping relationship between the design element and the surface of the geometric model, the system retrieves the 3D mesh topology structure of the target area in real time and generates the corresponding non-uniform rational spline curve; the control points of the non-uniform rational spline curve are associated with the mesh vertices of the geometric model in real time through weight factors. When the physical carrier undergoes deformation, causing the mesh vertices to shift, the control points are linked accordingly to ensure that the curvature of the design element is consistent with the curvature of the carrier surface.

[0013] Preferably, the physical-level visual rendering in step S23 includes: applying the calculated roughness and metallicity parameters to the shader model of the design element, and calculating the bidirectional reflection distribution function of the surface of the design element by combining the real-time captured light source direction information; and generating the projection effect of the design element onto the substrate based on the undulation state of the geometric model through shadow mapping technology, so that the virtual design element is completely integrated with the physical environment in terms of light and shadow logic.

[0014] Preferably, step S3 specifically includes: recording in real time the geometric adaptive curve parameters, material overlay properties, and dynamic lighting response logic generated during the design process; encapsulating the above data in a structured manner according to a predetermined standardized product model data exchange format or initial graphic exchange specification; embedding metadata for subsequent adaptive manufacturing into the encapsulated data, wherein the metadata includes spray trajectory compensation parameters for the deformed substrate.

[0015] A print reproduction design system integrating augmented reality technology, used to implement the above-mentioned method, includes: a multimodal perception module configured to collect depth and spectral information of a physical scene, the output of which is connected to a dynamic scene reconstruction module; a dynamic scene reconstruction module used to generate a 3D reconstruction model of geometry, materials, and environment through physically rendered material inverse calculation technology; a dynamic parametric design module connected to the dynamic scene reconstruction module, configured to receive interactive instructions and perform dynamic topology adaptive calculations to achieve real-time binding of virtual elements and physical substrates; and a closed-loop generation and output module configured to encapsulate dynamic parameters in the design process into standardized design units.

[0016] Preferably, the multimodal sensing module integrates a high-precision depth sensing unit and a multispectral imaging unit, which can synchronously capture the spatial coordinates and color features of the physical surface at a preset frequency.

[0017] Preferably, the dynamic parametric design module includes an interaction recognition unit, which uses computer vision algorithms to track the designer's gesture trajectory or eye-tracking focal plane and convert it into a three-dimensional coordinate sequence in virtual space.

[0018] Preferably, the closed-loop generation and output module includes a data conversion interface, which can convert non-uniform rational spline curves into path instructions that can be recognized by the manufacturing equipment, and perform dynamic coordinate correction based on the real-time pose data of the physical substrate.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention reconstructs augmented reality technology from a traditional output display end to an interactive design input end, and introduces a physically rendered material inverse calculation and dynamic topology adaptive mechanism.

[0020] 2. This invention automatically completes holographic registration of geometry, material, and lighting through real-time reverse engineering, eliminating the tedious process of image registration and feature extraction that relies on manual methods in traditional methods, and improving design efficiency.

[0021] 3. This invention uses a dynamic topology adaptive algorithm to enable design elements to sense the deformation of the physical carrier, solving the problems of perspective distortion and positional misalignment in the process of printing reproduction under non-ideal conditions, and ensuring the high fidelity and physical consistency of the design.

[0022] 4. The intelligent design unit output by this invention contains complete physical response logic and geometric parameters, which can not only be used for visual presentation, but also directly connect to subsequent digital twin and intelligent manufacturing processes, thus opening up the data link between design and production. Attached Figure Description

[0023] Figure 1This is a schematic diagram of the overall technical solution architecture proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the dynamic topology adaptive algorithm in this invention; Figure 3 This is a flowchart of the dynamic scene reconstruction logic based on physical rendering material inverse calculation in this invention. Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow between augmented reality interaction design and real-time binding of virtual elements in this invention; Figure 5 This is a logical flow diagram of the intelligent design unit closed-loop generation and manufacturing trajectory compensation in this invention; Figure 6 This is a schematic diagram of the overall technical architecture of the printed reproduction design system that integrates augmented reality technology proposed in this invention. Detailed Implementation

[0024] Example 1: Please refer to the appendix Figure 1 To be continued Figure 6 This embodiment provides a method for replicating printed materials using augmented reality technology. This method aims to address how to achieve high-precision digital replication designs in complex physical environments, such as when paper is bent, wrinkled, or under uneven lighting. This embodiment uses a design for restoring an ancient book with curvature and minute creases on its surface as an example to illustrate its application in detail.

[0025] Reference Figure 1 The method in this embodiment first performs step S1, which involves capturing the physical scene of the target printed material in real time through a multimodal sensing device, and then performing dynamic scene reconstruction based on physical rendering material inverse calculation technology to generate reconstruction data containing geometric dimensions, material dimensions and environmental dimensions.

[0026] In the specific implementation of this step, step S11 is first executed, using a depth sensing device and a spectral sensing device to collect surface information of the target printed material and its carrier. At the hardware level, the multimodal sensing device integrates a high-precision time-of-flight depth sensor operating at 60 Hz and an imaging unit with 16 spectral channels. The depth sensing device obtains the three-dimensional coordinates of each sampling point on the surface of the target printed material by transmitting a modulated pulse signal and receiving the time difference between the emitted and reflected signals. The spectral sensing device captures the surface radiance distribution in the visible and near-infrared bands, providing high-dimensional raw data for subsequent material analysis.

[0027] Then, step S12 is executed to extract a high-precision geometric model of the target printed surface using a point cloud reconstruction algorithm, in order to record the non-ideal geometric features of the carrier. The non-ideal geometric features recorded in this embodiment focus on the bending deformation of the paper, surface wrinkles, and the irregular topological structure of the physical carrier.

[0028] During point cloud reconstruction, the system employs a sampling strategy with an equal spacing of 0.5 mm to scan the physical surface. To ensure the model reflects subtle wrinkles, the system performs curvature analysis by calculating the angle between the normal vectors of adjacent sampling points to locate regions of geometric abrupt changes. When the rate of curvature change exceeds a preset threshold, the system automatically increases the sampling density in the corresponding local area, constructing a dynamic mesh model that reflects the deformation state of the physical entity in real time. This dynamic mesh model consists of tens of thousands of triangular facets, and the displacement data of each vertex is dynamically updated according to the real-time state of the physical carrier.

[0029] Next, step S13 is executed: by analyzing the interaction between incident and reflected light, the basic color, roughness, and metallicity parameters of the printed surface are calculated in reverse. The core of this process lies in establishing a reflection distribution function model based on micro-surface theory. The system uses the radiance data acquired by the spectral sensing device as input and employs iterative optimization logic to solve the surface reflection equation.

[0030] The system simulates the reflection and refraction paths of light on microscopic geometric structures, continuously correcting the surface's physical parameters by comparing the observed brightness distribution with the model's predictions. Ultimately, the system can extract the intrinsic physical properties of the printed surface, including the fundamental reflectivity reflecting the material's essential characteristics, the roughness reflecting the surface's microscopic smoothness, and the metallicity parameters reflecting the properties of dielectrics and conductors. This processing method eliminates the interference of ambient light sources on the material's color characterization, ensuring the consistency of material data acquired under different lighting conditions.

[0031] Execute step S14 to estimate the ambient light occlusion information and light source direction information in the physical scene in real time, and complete the extraction of global illumination information.

[0032] In one specific embodiment, the extraction of ambient light occlusion information and the estimation of light source direction information in step S14 are achieved through the following technical means.

[0033] Ambient light occlusion estimation: The spherical harmonic projection method uses third-order (L=3) spherical harmonic basis functions to approximate the illumination distribution in space. For any point on the surface of the reconstructed geometric model... Its ambient light shading value Defined as the ratio of the indirect illumination intensity received at that point to the intensity under completely unobstructed conditions. The dynamic scene reconstruction module first at the point... A set of uniformly distributed sampling directions is generated on the tangent plane. Each direction corresponds to a spherical harmonic basis function. Then, using a ray casting algorithm, it is determined from the point... Along direction Does the emitted ray intersect with other parts of the geometric model? Let the intersection function be... The value is 1 (with occlusion) or 0 (without occlusion). This represents the ambient light occlusion value. The following is obtained by weighted averaging of occlusion conditions across all sampling directions: ; in, It is a point The normal direction, coefficient Calculated using hemispherical integrals: ; By solving the above integral, the system generates an ambient occlusion layer that corresponds one-to-one with the vertices of the geometric model, with values ​​ranging from 0 (complete occlusion) to 1 (no occlusion).

[0034] Light source direction estimation: The light source direction extraction is achieved by analyzing the distribution characteristics of the highlight regions on the physical surface. The system first uses image processing algorithms (such as threshold segmentation) to identify the clusters of pixels with the highest brightness in the multispectral image, and denotes the three-dimensional spatial point set corresponding to these pixels as... For each highlight point Its surface normal Given. According to the law of reflection, the direction of the light source. ; With line of sight Camera pointing point (vector) and normal The following relationship must be satisfied: ; The system performs cluster analysis on all candidate light source directions calculated from the specular highlights, and selects the center of the direction cluster with the highest density as the main light source direction. When multiple light sources are present, the system analyzes the highlight characteristics of different regions in the image to identify multiple directional clusters and records the relative intensity of each light source. The intensity is determined by the average brightness value of the highlight area corresponding to the light source.

[0035] The process of extracting ambient light occlusion information includes: projecting spherical harmonic functions based on the geometric model reconstructed in step S12, and generating an ambient light occlusion layer by calculating the proportion of each grid point in space that is occluded by the surrounding geometric structure. For example, in the depths of paper folds, the occlusion proportion is higher, and the light intensity will be correspondingly reduced.

[0036] The extraction of light source direction information is achieved by analyzing the distribution characteristics of the highlight areas on the physical surface. The system retrieves the cluster of pixels with the highest brightness in the image in real time, and combines it with the surface normal vector data to calculate the vector coordinates of the main light source in three-dimensional space, thereby determining the precise position and illumination angle of the light source.

[0037] Reference Figure 2 and Figure 4 After the scene reconstruction is completed, step S2 is executed: an interactive dynamic parameterized design space is constructed based on the reconstruction data, and virtual design elements are bound to the reconstruction data in real time and dynamically adjusted for topology adaptation.

[0038] First, step S21 is executed to establish parametric design anchor points in the augmented reality view and recognize the designer's interaction commands. The designer wears an augmented reality headset and can see a digital twin model that perfectly matches the physical entity in the virtual view. The system uses computer vision algorithms to track the designer's gesture trajectory. When the designer's fingertips move above the physical printed surface, the system projects their three-dimensional spatial trajectory onto the mesh surface of the geometric model, establishing a series of design anchor points with physical properties.

[0039] Then, step S22 is executed: design elements are generated in the virtual space based on the interactive instructions, and the design elements are mapped to the geometric model using a dynamic topology adaptive algorithm. When the designer draws a decorative curve on the curved paper surface, the dynamic topology adaptive algorithm will retrieve the 3D mesh topology structure of the target area in real time.

[0040] The system does not simply generate two-dimensional lines, but rather corresponding non-uniform rational spline curves. The control points of these non-uniform rational spline curves are linked in real-time to the mesh vertices of the geometric model through weighting factors. This means that when the physical carrier undergoes further deformation due to external forces, causing displacement of the mesh vertex coordinates, the control points of the spline curve will move in tandem according to a preset weight ratio. In this way, the curvature of the design elements can always maintain a high degree of consistency with the real-time curvature of the carrier surface, eliminating perspective distortion.

[0041] Step S23 involves physically based visual rendering of the mapped design elements according to the material dimension parameters and global illumination information. This process applies the roughness and metallicity parameters calculated in step S13 to the shader model of the design elements. Combining the real-time captured light source direction information from step S14, the system calculates the bidirectional reflection distribution function of the design element's surface. To enhance realism, the system also employs shadow mapping technology, generating the projection effect of the design elements onto the base surface based on the undulations of the geometric model. For example, when the virtual gold line is on a raised part of a paper fold, its specular reflection is more intense; while when it is in a recessed part, it is darkened by the ambient light occlusion layer, allowing the virtual design elements to fully integrate with the physical environment in terms of light and shadow logic.

[0042] In one specific embodiment, the process of reversely calculating the basic color, roughness, and metallicity parameters of the printed surface in step S13 adopts the Cook-Torrance reflection model based on micro-surface theory and combines it with a nonlinear least squares optimization algorithm to solve for the parameters.

[0043] The reflection model is defined as follows: The BRDF (Radiation Distribution Function) model decomposes surface reflection into a weighted sum of diffuse and specular reflection components. For a given incident direction... and launch direction BRDF value Defined as: ; in, It is the diffuse reflectance coefficient (related to the base color). It is the specular reflectance coefficient (related to metallicity). This is the distribution function of the microsurface normal, using the GGX distribution, used to describe the statistical distribution of the microsurface normal. Its expression is: ; in, It is a roughness parameter. It is the macroscopic surface normal. It is a half-angle vector (the angle bisector of the incident and outgoing directions). This is the Fresnel term, describing the reflectivity of light at different angles, and is approximated using the Schlick approximation: ; in, It is the reflectance when the normal is incident (related to the base color). It refers to the direction of the line of sight. It is a geometrical attenuation term that describes the self-occlusion between micro-surfaces and uses the Smith joint occlusion function.

[0044] Optimization objective function: The iterative optimization logic aims to minimize the difference between the observed radiance and the model prediction. For the... There are several observation points, and the direction of incident light is assumed to be... The direction of observation is The radiance value captured by the multispectral imaging unit is Then optimize the objective function. Defined as the root mean square error for all observation points: ; in, It is the material parameter vector to be optimized. It is the number of effective observation points. From direction The incident ambient light intensity (estimated by step S14). Let represent a hemispherical space. The integral term represents the predicted outgoing radiance of the model under given illumination and material parameters.

[0045] Iterative optimization solution: The system employs the Levenberg-Marquardt algorithm to solve the aforementioned nonlinear least squares problem. In each iteration, the parameter update amount... This is obtained by solving the following system of linear equations: ; in, It is a Jacobian matrix, whose elements , It is the first The residuals at each observation point It is the damping factor. It is the identity matrix. Iteration continues until the parameter change between two consecutive iterations is less than a preset threshold. (like ), or objective function The change is less than the preset threshold The final converged result is... That is, the intrinsic physical properties of the surface of the printed matter, wherein Corresponding to basic reflectivity, Corresponding roughness value, Corresponding metallicity parameters.

[0046] Reference Figure 5 Finally, step S3 is executed to generate a closed loop based on the dynamic parameterized data during the design interaction process, and to encapsulate and output an intelligent design unit that conforms to the computer-aided design standard.

[0047] In this step, the system records all core data generated during the design process in real time, including control point parameters of the geometric adaptive curve, physical properties of the material overlay, and dynamic lighting response logic. The system structures and encapsulates this data according to standardized initial graphic exchange specifications, ensuring seamless readability by mainstream industrial software. Most importantly, metadata for subsequent adaptive manufacturing is embedded in the encapsulated data. This metadata includes spray trajectory compensation parameters for deformed substrates. In future manufacturing stages, the spraying equipment can adjust the nozzle's movement path and spray pressure in real time based on these compensation parameters.

[0048] Example 2: This example, based on Example 1, further extends and refines the technology for the imitation design of irregularly shaped packaging containers. In this scenario, the printing medium is no longer flat paper, but a plastic or metal bottle with complex curvature.

[0049] When performing step S1, which involves collecting data using a multimodal sensing device, a polarization filter unit is added to the multimodal sensing module to address the container's high reflectivity. By capturing spectral information at different polarization angles, the system can eliminate strong reflective highlights on the surface and more accurately calculate the surface's basic color data.

[0050] During the point cloud reconstruction in step S12, a multi-resolution mesh generation strategy was introduced to address the topology of the heterogeneous bottle. For edge regions with drastic changes in bottle curvature, ultra-fine sampling with a step size of 0.1 mm was used; while for flat middle regions, conventional sampling with a step size of 1.0 mm was employed. This non-uniform sampling strategy, while maintaining geometric accuracy, reduced the computational load of subsequent topology adaptive calculations and improved the smoothness of real-time interaction.

[0051] When executing the dynamic topology adaptive algorithm in step S22, a constraint term is added to the algorithm considering the rigidity of the packaging container. For incompressible regions on the container, the system limits the displacement range of the control points by setting a rigidity factor when establishing the association between the spline curve control points and the mesh vertices. This ensures that even when the sensor captures slight jitter, the position of the design elements on the container surface remains robust and does not exhibit significant drift.

[0052] In one specific embodiment, the dynamic topology adaptive algorithm in step S22 adopts a linear hybrid skinning technique based on centroid coordinates to achieve real-time association and linkage between the control points of non-uniform rational spline curves (NURBS) and the vertices of the geometric model mesh.

[0053] Initial mapping establishment: When a designer specifies a location on the physical carrier surface to generate a design element, the system first determines the triangular facet (mesh vertex) where that location is located. Let the coordinates of the barycenter of this position within the triangular facet be ( ). ,satisfy and The coordinates of this location in three-dimensional space. It can be represented as: ; The system then generates control points for the NURBS curve at that location. Its initial position and The control point coincides. The system records this control point. With the vertex of the triangular face The relationships between them are stored, and the centroid coordinates are stored. As a weighting factor.

[0054] Linked update mechanism: When the physical carrier deforms, causing the mesh vertices to shift, the vertex is set to... The new positions are respectively Based on pre-stored relationships and weighting factors, the system updates control points in real time. new location : ; For all control points of the NURBS curve, the system performs the above update operations in parallel. This linkage mechanism based on barycentric coordinate interpolation ensures that design elements can undergo smooth and continuous deformation as the physical carrier deforms, and their curvature always remains consistent with the curvature of the carrier surface. When the physical carrier breaks or folds, causing the topology of the original triangular facet to be destroyed, the system triggers topology reconstruction: by identifying the broken edges, the association relationships of the control points are reassigned to the newly generated, spatially nearest neighbor triangular facet after deformation, and its barycentric coordinates are recalculated.

[0055] During the visual rendering process in step S23, the system specifically enhanced the refraction and reflection representation of the metallic packaging. By introducing a high dynamic range environment map and combining it with the real-time lighting vector extracted in step S14, the system achieved complex multi-reflection effects on the surface of the virtual design elements, enabling them to achieve photorealistic visual presentation on the metal substrate.

[0056] During the closed-loop generation of step S3, the intelligent design unit of the package not only includes geometric parameters but also additional metadata for wall thickness compensation specific to injection molding or printing processes. This data guides the manufacturing end to automatically optimize ink deposition thickness when handling irregular bottle shapes, achieving a balance between visual appeal and physical durability.

[0057] Example 3: This example focuses on the imitation design of printed textiles. The characteristics of this type of scenario are that the physical carrier has extremely high flexibility and complex texture structure, and large-scale nonlinear deformation often occurs during the interaction process.

[0058] During step S1, the multimodal sensing device operates at a high sampling rate of 120 Hz to capture the instantaneous state of the fabric during movement. The material inverse calculation process in step S13 is optimized for the translucent properties of textiles. The system employs a subsurface scattering model to simulate the propagation of light within the fibers, calculating the fabric's transmittance parameters. This allows the system to simulate the color blending effect between design elements and fabric texture when overlaying virtual designs, avoiding a simple overlay effect.

[0059] In step S22, the dynamic topology adaptive algorithm employs skinning logic based on physical simulation. The system treats the mesh vertices of the fabric as point masses with mass and elastic modulus, and the control points of the spline curves are connected to the nearest mesh vertex via virtual springs. When the fabric is subjected to significant crumpling or folding, the design elements can deform naturally with the stretching and compression of the fibers, just as if they were sewn onto the fabric, maintaining extremely high physical consistency.

[0060] During step S3, the intelligent design unit generated by the system includes displacement mapping data of the fabric texture. This means that the output design results can be directly applied to advanced garment simulation software, restoring a realistic sense of texture through displacement parameters. The output metadata includes proportional adjustment parameters for the fabric stretch rate, ensuring that the pattern can compensate for deformation errors caused by fabric pre-stretching during the actual printing and dyeing process.

[0061] Example 4: This example details the printed matter replication design system integrating augmented reality technology proposed in this invention. This system is used to execute the methods described in the above examples. (Refer to...) Figure 6 The system's hardware architecture and software modules are deeply integrated to achieve a closed loop across the entire chain from perception to design.

[0062] The system includes a multimodal sensing module. This module integrates a high-precision depth sensing unit and a multispectral imaging unit. The depth sensing unit utilizes laser speckle structured light technology to acquire 3D point cloud information of the physical scene within microseconds. The multispectral imaging unit is equipped with a narrowband filter array, capable of synchronously capturing the spatial coordinates and color features of the physical surface at a frequency of 30 frames per second. These two sub-units ensure precise spatiotemporal alignment of the depth map and the spectral map through a timestamp synchronization mechanism.

[0063] The output of the multimodal perception module is connected to the dynamic scene reconstruction module. This module incorporates a high-performance graphics processing cluster to execute physically based rendering material inverse calculation algorithms. Through parallel computation, the module transforms the acquired raw light and depth signals into a 3D reconstruction model with geometric, material, and environmental features. The environment estimation unit uses spherical harmonic coefficients to characterize the global illumination field in real time, providing an optical baseline for subsequent rendering.

[0064] The output of the dynamic scene reconstruction module is connected to the dynamic parametric design module. This module includes an interaction recognition unit and a topology calculation unit. The interaction recognition unit uses computer vision algorithms to track the designer's gestures or eye-tracking focal planes, converting them into a sequence of three-dimensional coordinates in virtual space. The topology calculation unit executes the generation and binding logic of non-uniform rational spline curves in real time, dynamically adjusting the geometric properties of design elements based on the mesh changes of the reconstructed model. The dynamic parametric design module also includes a physically based rendering engine, responsible for real-time synthesis of inversely calculated material parameters, ambient lighting, and virtual elements to generate an augmented reality view.

[0065] The system includes a closed-loop generation and output module. This module is equipped with a data conversion interface, responsible for converting non-uniform rational spline curves into path instructions recognizable by manufacturing equipment. The module has a built-in coordinate correction engine that performs differential compensation on the output geometric data based on the pose drift trajectory of the physical substrate during the design process. The final output intelligent design unit contains complete manufacturing metadata and is interfaced with subsequent CNC machining or digital twin platforms via a standardized industrial bus interface.

[0066] The workflow of this system is as follows: the multimodal perception module continuously scans the physical environment and transmits the high-dimensional data stream to the dynamic scene reconstruction module for physical attribute stripping; designers interact and create on the reconstructed digital twin, and the dynamic parametric design module ensures that virtual elements achieve zero distortion fitting on complex physical carriers; finally, the closed-loop generation module outputs the design results containing physical response logic in a structured manner, completing the entire process from reality capture to digital reconstruction.

[0067] In this embodiment, the depth sensing unit of the multimodal perception module has a measurement accuracy of less than 1 millimeter, and the spectral imaging unit can distinguish minute color differences that are imperceptible to the naked eye. This high-precision hardware support, combined with the algorithm logic based on micro-surface theory in the dynamic scene reconstruction module, enables the system to identify roughness differences on the surface of printed materials as small as 20 micrometers. This extraction of deep physical features lays a solid foundation for the subsequent realization of physically based visual rendering.

[0068] In one specific embodiment, the physically based visual rendering in step S23 employs a physically based shading model and real-time shadow mapping technology.

[0069] Bidirectional reflection distribution function calculation: For any point on the surface of the virtual design element, its shader model adopts the same Cook-Torrance BRDF model as in step S13. The final color of this point... It consists of three parts: ambient light, diffuse light, and specular light. ; Among them, ambient light component Ambient occlusion map extracted in step S14 and global ambient color Joint decision: ; Diffuse reflection component Based on the Lambert model, considering the direction of the main light source and surface normal : ; Specular reflection component Based on the roughness calculated in step S13 and Fresnel : ; Shadow Mapping: For each light source, the system uses shadow mapping technology to generate a shadow map. First, using the light source position as the viewpoint, the depth map of the geometric model is rendered to obtain the shadow map. When rendering virtual design elements, for points on the element's surface... Transform its coordinates to the light source space to obtain texture coordinates. And compare their depth values Depth value of the corresponding pixel in the shadow map .like ( If the distance is the depth offset, then the point is in shadow. For multi-light source scenes, the system generates a shadow map independently for each light source and then superimposes the contributions from all shadow layers. ; in, The total number of light sources It is a point Compared to the first The visibility of each light source (0 indicates in shadow, 1 indicates illuminated). This refers to the intensity of the light source. The final pixel color. exist Multiply by the shading factor To achieve physical-level shadow blending under multiple light sources.

[0070] In the dynamic parametric design module, the recognition latency for gesture interactions is controlled to within 15 milliseconds, enhancing the designer's immersive experience. The topology calculation unit maintains real-time updates even when processing complex models containing over 100,000 faces. When a designer modifies the properties of a design element in the 3D view, the system automatically recalculates the shadow and highlight distribution of the affected area, ensuring that each frame conforms to the fundamental laws of physical optics.

[0071] The data conversion interface of the closed-loop generation and output module supports multiple industry standards such as STEP, IGES, and STL. During data encapsulation, the system samples the spline curve and converts it into a G-code sequence that can be directly executed by CNC equipment. By embedding compensation parameters in the data packet, the system solves the problem of decreased machining accuracy caused by carrier positioning deviation or deformation.

[0072] Example 5: This example further explores the robustness of the invention under extreme lighting and complex material conditions. In this example, the target printed material is placed in a semi-open indoor environment with multiple point light sources and strong diffuse reflection interference.

[0073] During step S14, the system analyzes multiple local highlight areas to identify three main light sources in the physical scene. The ambient occlusion extraction process not only considers static geometric occlusion but also introduces dynamic self-occlusion calculation. As the physical carrier moves, the system updates the ratio of the illuminated area of ​​each fold in real time, generating continuous and smooth tonal changes in the ambient occlusion layer.

[0074] During the material inverse calculation in step S13, a multi-layer reflection model was introduced for high-gloss printed materials with coating layers. In addition to basic micro-surface reflection, the system also calculated the Fresnel reflection contribution of the transparent coating layer surface. By analyzing the energy loss of light at the interface of media with different refractive indices, the system accurately separated the base color of the printing ink from the specular highlights of the surface varnish. This processing logic ensures that the virtual elements superimposed in subsequent designs can present a texture level completely consistent with the original artwork.

[0075] In the interaction process of step S22, to address the complex projection issues caused by multiple light sources, the dynamic topology adaptive algorithm adds a light intensity weighting factor. When a design element is located in an overlapping area illuminated by multiple light sources, the system calculates the transparency and edge blur of multiple shadow layers based on the distance and intensity vectors of each light source, and then physically overlays and blends these layered shadows. This meticulous processing method gives the virtual design a strong sense of spatial presence, eliminating the floating sensation common in existing AR technologies.

[0076] In the output stage of step S3, the intelligent design unit of this embodiment additionally includes photosensitive parameter metadata. This data records the reflection characteristics of the material under different incident angles and can be used for subsequent digital quality inspection. By comparing the reflection distribution of the finished product and the design model under the same lighting conditions, automated imitation accuracy evaluation can be achieved.

[0077] Example 6: This example focuses on describing the self-healing and self-alignment mechanism of the present invention in the dynamic interaction process, which is the core technical feature to ensure that the system maintains high precision during long-term operation.

[0078] In one specific embodiment, the self-healing and self-alignment mechanism employs point cloud registration technology based on the Iterative Closest Point (ICP) algorithm, combined with a spatial correlation matrix to achieve rapid relocation of the geometric model.

[0079] Displacement detection: The multimodal sensing module uses frequency (e.g., 60Hz) Continuously capture point cloud data of the physical scene. Let the current time be... The captured feature point cloud is The vertex set of the geometric model reconstructed in the previous moment is The system first performs a nearest neighbor search to... Each point in exist Find the corresponding point If the average distance of all corresponding point pairs Exceeding the preset threshold If the distance is 5 mm, the repositioning mechanism will be triggered.

[0080] Rigid body transformation solution: The system finds the optimal rotation matrix by solving the following least squares problem. Translation vector This makes the transformed model vertex set With the current point cloud The error between corresponding point pairs is minimized: ; in, It is the number of point pairs participating in registration. yes exist The corresponding point indices in the dataset. This problem can be solved using the Singular Value Decomposition (SVD) method. First, calculate the centers of the two point sets. and Then construct the covariance matrix. : ; right Perform SVD decomposition: The optimal rotation matrix is The optimal translation vector is .

[0081] Spatial Intent Matrix and Model Update: The system will solve for the following: and Stored as a spatial incidence matrix. Subsequently, for all vertices of the geometric model... Apply rigid body transformation to update and obtain a new set of model vertices. : ; Simultaneously, all virtual design elements bound to this geometric model (such as control points for NURBS curves) also undergo the same rigid body transformation to ensure precise alignment between the design anchor points and the physical surface. The entire repositioning process is completed within milliseconds, guaranteeing the continuity and stability of the interaction.

[0082] During system operation, the multimodal perception module monitors the displacement vectors of point cloud feature points in real time, as the physical carrier may be frequently touched or moved. When the detected global displacement exceeds a preset step size, the dynamic scene reconstruction module triggers rapid relocation logic. This logic does not re-execute a complete reconstruction, but rather aligns the existing geometric model with the newly captured feature point cloud through a rapid rigid body transformation using a spatial correlation matrix. This process is completed within milliseconds, ensuring the positional stability of the designed anchor points on the physical surface.

[0083] When the physical carrier undergoes irreversible and permanent large deformation, the dynamic topology adaptive algorithm in step S22 will initiate topology reconstruction. The system will automatically identify broken edges in the mesh and redistribute the control points of the spline curves to new mesh regions. This adaptive capability makes the design process highly fault-tolerant, allowing even non-professionals operating in unstable environments to obtain a continuous and consistent design experience.

[0084] In the physically-level visual rendering of step S23, the system also introduces motion blur compensation. When the image stream acquired by the sensing device becomes blurred due to rapid movement, the rendering engine estimates the compensation parameters based on the motion vectors and performs synchronous convolutional blurring on the virtual elements. This ensures that the virtual information and the background physical environment remain synchronized in visual clarity during dynamic observation.

[0085] During the output encapsulation in step S3, the system generates a log metadata document detailing the interaction process. This log records in detail the flow of all deformation parameters of the design element from its initial state to its final form. For complex replication tasks, this data stream can be used to analyze the mechanical properties of the substrate material, providing a scientific basis for subsequent process selection.

[0086] Through the detailed descriptions of the above embodiments, it can be seen that the present invention constructs a complete closed loop from the physical world to the digital space and back to physical manufacturing through a series of technical means such as multimodal perception, physically rendered material inverse calculation, dynamic topology adaptation, and closed-loop encapsulation output. This solution not only improves the design efficiency of printed reproduction and realizes automated workflow with zero manual registration, but also ensures the high fidelity and manufacturability of the design solution under non-ideal environments through deep physical property analysis and adaptive algorithms.

[0087] In the technical solution of this invention, all logical operations are based on pure textual engineering descriptions. For example, when performing material inverse calculation, the system obtains the radiance values ​​collected by the sensing device, uses them as the basis for measuring the surface reflectivity, and derives the surface roughness parameters based on the radiance attenuation ratio at different observation angles. This process completely eliminates any interference from mathematical formulas or symbols, and is achieved through the causal logic describing physical phenomena and the flow of data processing.

[0088] In the spline curve generation logic, the system identifies the displacement vectors of the grid vertices and assigns these displacement vectors to the control points of the spline curve according to preset weighting coefficients, causing the spatial coordinates of the control points to shift accordingly. This descriptive method ensures that those skilled in the art can clearly understand the physical essence and implementation path of the algorithm without relying on formulas.

[0089] The system architecture design of this invention fully considers the stability and scalability requirements in industrial applications. The high-frequency sampling and multispectral resolution capabilities of the multimodal perception module provide a high-quality data source for all subsequent logical judgments; the physical-level inverse calculation capability of the dynamic scene reconstruction module ensures the authenticity of the digital twin; and the dynamic parametric design module lowers the technical threshold of design through innovative AR interaction methods.

[0090] The final output intelligent design unit, a key achievement of this invention, breaks down the barriers between traditional visual and manufacturing models. By embedding deformation compensation metadata, this invention transforms the design outcome from a mere image on a screen into an engineering blueprint capable of guiding physical manufacturing. This holds irreplaceable technological value for driving the printing industry towards intelligent, digital, and personalized transformation.

[0091] In summary, this invention, through in-depth analysis and parametric reconstruction of the properties of the physical world, successfully eliminates the trade-off between efficiency and accuracy present in existing technologies. Whether in the demanding needs of ancient book restoration or in the large-scale customization scenarios of the modern packaging industry, this invention provides robust, efficient, and high-precision technical support.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for replicating printed materials using augmented reality technology, characterized in that, Includes the following steps: Step S1: Collect surface information of the target printed matter and its carrier using a multimodal sensing device. Extract a high-precision geometric model of the target printed matter surface based on a point cloud reconstruction algorithm to record the non-ideal geometric features of the carrier. Analyze the interaction between incident and reflected light and establish a reflection distribution function model based on micro-surface theory. In this way, calculate the basic color, roughness, and metallicity parameters of the printed matter surface in reverse. At the same time, estimate the ambient light occlusion information and light source direction information in the physical scene in real time to complete the generation of dynamic scene reconstruction data including geometric, material, and environmental dimensions. Step S2: Establish parametric design anchor points in the augmented reality view and identify the designer's interaction commands. Based on the interaction commands, generate design elements in the virtual space. Use a dynamic topology adaptive algorithm to map the design elements to the geometric model, generate corresponding non-uniform rational spline curves, and associate the control points of the non-uniform rational spline curves with the mesh vertices of the geometric model in real time through weight factors. When the physical carrier deforms and causes the mesh vertices to shift, the control points will move accordingly. According to the material dimension parameters and global illumination information, perform physical-level visual rendering on the mapped design elements to construct an interactive dynamic parametric design space in which virtual design elements are bound to the reconstructed data in real time and dynamically adjusted in topology. Step S3: Record the geometric adaptive curve parameters, material overlay properties, and dynamic lighting response logic generated during the design process in real time. Encapsulate the above data in a structured manner according to the predetermined standardized product model data exchange format. Embed metadata, including spray trajectory compensation parameters for deformed substrates, for subsequent adaptive manufacturing into the encapsulated data. Output an intelligent design unit that conforms to computer-aided design standards.

2. The method for replicating printed materials using augmented reality technology according to claim 1, characterized in that, The non-ideal geometric features recorded in step S1 include the bending deformation of the paper, surface wrinkles, and the irregular topological structure of the physical carrier. During the extraction of the geometric model by the point cloud reconstruction algorithm, the system uses a sampling strategy with a preset step size and equal spacing to scan the physical surface. It performs curvature analysis by calculating the angle between the normal vectors of adjacent sampling points to locate regions of geometric abrupt change. When the rate of curvature change is found to exceed a preset threshold, the sampling density is automatically increased in the corresponding local area to construct a dynamic mesh model composed of triangular facets that reflects the deformation state of the physical entity in real time. The displacement data of the vertices of the dynamic mesh model are dynamically updated according to the real-time state of the physical carrier.

3. The method for replicating printed materials using augmented reality technology according to claim 1, characterized in that, The process of reversely calculating the basic color, roughness, and metallicity parameters of the printed surface in step S1 is as follows: the radiance data acquired by the multimodal sensing device is used as input, the surface reflection equation is solved using iterative optimization logic, the reflection and refraction path of light on the micro-geometric structure is simulated, the physical parameters of the surface are corrected by comparing the deviation between the observed brightness distribution and the model prediction value, and the intrinsic physical properties of the printed surface are extracted. The intrinsic physical properties include the basic reflectivity reflecting the essential characteristics of the material, the roughness value reflecting the micro-smoothness of the surface, and the metallicity parameter reflecting the properties of the dielectric and conductor, eliminating the interference of ambient light on the color characterization of the material itself.

4. The method for replicating printed materials using augmented reality technology according to claim 1, characterized in that, The process of estimating ambient light occlusion information and light source direction information in real time in step S1 is as follows: based on the reconstructed geometric model, a spherical harmonic function projection is performed, and an ambient light occlusion layer is generated by calculating the proportion of each grid point in space that is occluded by the surrounding geometric structure. The extraction of the light source direction information is achieved by analyzing the distribution characteristics of the highlight area on the physical surface. The system searches for the cluster of pixels with the highest brightness in the image in real time, and combines the surface normal vector data to calculate the vector coordinates of the main light source in three-dimensional space, thereby determining the precise position and illumination angle of the light source.

5. The method for replicating printed materials using augmented reality technology according to claim 1, characterized in that, The specific implementation logic of the dynamic topology adaptive algorithm in step S2 is as follows: when establishing a mapping relationship between the design element and the surface of the geometric model, the system retrieves the three-dimensional mesh topology of the target area in real time and generates the non-uniform rational spline curve. The control points of the non-uniform rational spline curve are linked in real time with the grid vertices of the geometric model through a weight factor. When the physical carrier deforms due to external force, causing the coordinates of the grid vertices to shift, the control points are linked accordingly according to the preset weight ratio to ensure that the curvature of the design element is consistent with the real-time curvature of the carrier surface and to eliminate perspective distortion.

6. The method for replicating printed materials using augmented reality technology according to claim 1, characterized in that, The physical-level visual rendering process in step S2 is as follows: the calculated roughness and metallicity parameters are applied to the shader model of the design element, and the bidirectional reflection distribution function of the surface of the design element is calculated in combination with the real-time captured light source direction information. By using shadow mapping technology, the projection effect of design elements onto the base surface is generated based on the undulation state of the geometric model. Multiple local highlight areas are identified based on the number of main light sources in the physical scene. The transparency and edge blur of multiple shadow layers are calculated based on the distance and intensity vector of each light source and then superimposed and blended, so that the virtual design elements are integrated with the physical environment in terms of light and shadow logic.

7. The method for replicating printed materials using augmented reality technology according to claim 1, characterized in that, The process of outputting the intelligent design unit in step S3 is as follows: the control point parameters of the geometric adaptive curve, the physical properties of the material overlay layer, and the dynamic lighting response logic generated during the design interaction process are structurally encapsulated according to the standardized initial graphic exchange specification. The metadata embedded in the packaging data also includes wall thickness compensation metadata for injection molding or printing processes and proportional adjustment parameters for fabric stretching rate. These are used to guide the manufacturing end to automatically optimize the ink buildup thickness or compensate for deformation errors caused by fabric pre-stretching when handling irregular carriers, thereby achieving physical precision correction in the manufacturing process.

8. The method for replicating printed materials using augmented reality technology according to claim 1, characterized in that, The method also includes self-healing and self-alignment steps in the dynamic interaction process: the multimodal sensing device monitors the displacement vector of the feature points of the point cloud in real time. When the global displacement exceeds the preset step size, the existing geometric model and the newly captured feature point cloud are quickly aligned by rigid body transformation through the spatial correlation matrix. When the physical carrier breaks or folds, resulting in large deformation, the system automatically identifies the broken edge in the mesh and reassigns the control points of the spline curve to a new mesh area. When the image stream acquired by the sensing device becomes blurred due to movement, the virtual elements are synchronously subjected to convolutional blurring processing based on the motion vector to estimate compensation parameters, thus maintaining the synchronization of visual clarity between the virtual information and the background physical environment.

9. A printed reproduction design system integrating augmented reality technology, characterized in that, The method for implementing any one of claims 1 to 8 includes: a multimodal sensing module configured to simultaneously acquire depth information and spectral information of a physical scene using a depth sensing unit and a multispectral imaging unit, wherein the depth sensing unit acquires three-dimensional point cloud information using laser speckle structured light technology, and the multispectral imaging unit captures the spatial coordinates and color features of a physical surface through a narrowband filter group; The dynamic scene reconstruction module is connected to the multimodal perception module and is configured to convert the original light signal and depth signal into a three-dimensional reconstruction model with geometric, material and environmental features based on the physical rendering material inverse calculation technology, and to use the environment estimation unit to characterize the global illumination field through the spherical harmonic function coefficients. A dynamic parametric design module, connected to the dynamic scene reconstruction module, is configured to receive interactive instructions and perform dynamic topology adaptive calculations. The dynamic parametric design module includes an interactive recognition unit and a topology calculation unit. The interactive recognition unit converts the tracked gesture trajectory or eye-tracking focal plane into a three-dimensional coordinate sequence in virtual space. The topology calculation unit executes the generation and binding logic of non-uniform rational spline curves. The closed-loop generation and output module is connected to the dynamic parameterized design module and is configured to encapsulate the dynamic parameters in the design process into standardized design units containing manufacturing metadata. The closed-loop generation and output module has a built-in coordinate correction engine, which is used to perform differential compensation on the output geometric data based on the pose drift trajectory of the physical substrate in the design process.

10. The printed matter replication design system integrating augmented reality technology according to claim 9, characterized in that, The multimodal sensing module integrates a polarization filter unit, configured to capture spectral information at different polarization angles to eliminate strong reflective highlights on the surface; The dynamic scene reconstruction module adopts a multi-resolution mesh division strategy for irregularly shaped carriers, increasing the sampling frequency in edge regions with drastic curvature changes and decreasing the sampling frequency in flat regions. The topology calculation unit in the dynamic parametric design module has a constraint term for the rigid carrier, which limits the displacement range of spline curve control points by setting a rigidity factor. The closed-loop generation and output module is equipped with a data conversion interface, which is used to convert non-uniform rational spline curves into path instructions executed by the manufacturing equipment, and to perform dynamic coordinate correction based on the real-time pose data of the physical carrier.

Images