Digital technology assisted physical creation method, device, product and medium

Abstract

The invention discloses a method, equipment, a product and a medium for digital technology assisted physical creation, and relates to the field of digital technologies. The method comprises the steps of collecting a multi-view image of a physical creation entity and performing three-dimensional reconstruction to generate current state data; comparing the data with a preset digital target model to generate deviation data; generating a calibration visual instruction based on the deviation data, and displaying a calibration mark on a front-end display interface to guide a user; and collecting interaction data including confirmation or denial of the calibration instruction by the user. According to the method, the creation deviation can be dynamically calibrated, the conformity of the physical entity and the digital target model is improved, meanwhile, the final decision-making right of a creator is reserved through user interaction, interaction feedback of man-machine cooperation in the physical creation process is achieved, and then the physical creation efficiency of the creator is improved.

Description

Technical Field

[0001] This application relates to the field of digital technology, and in particular to a method, device, product, and medium for digital technology-assisted physical creation. Background Technology

[0002] In recent years, with the development of digital technology, using projection, augmented reality (AR), and other methods to assist humans in creating and manipulating the physical world has become an important direction for improving efficiency and accuracy. Whether it's industrial assembly, artistic painting, model building, or handicrafts, digital assistive devices can provide operators with intuitive visual guidance.

[0003] Currently, mainstream digital assistive devices can be mainly divided into the following two categories: The first category is guidance devices based on static projection. These devices project digital information such as design drawings, outlines, or operating procedures directly onto a physical workbench or creative entity. For example, they project line drawings onto a canvas to aid in outlining, or project the mounting positions of components onto a circuit board to aid in soldering. The advantages of this technology are its intuitiveness and low cost. However, its fatal flaw lies in the unidirectional nature of information transmission. The projection device is only responsible for "outputting" guidance information, but is completely "blind" to the user's actual operational behavior in the physical world. It cannot sense whether the user's brush has deviated from the outline, nor can it determine whether the user has installed the wrong component. The information flow is open-loop; there is a lack of necessary feedback mechanisms between the device and the operator, and any operational deviations cannot be detected and corrected by the device.

[0004] The second category is guidance devices based on traditional augmented reality. These devices overlay 3D models or virtual instructions onto the real environment through AR glasses or mobile device screens. For example, when performing complex mechanical repairs, AR devices can highlight screws that need tightening or play animations of the next step. Compared to static projection, AR devices offer richer and more three-dimensional interactive forms. However, existing AR guidance devices still fundamentally fail to solve the aforementioned open-loop problem. The displayed digital information is usually preset and programmed; it methodically demonstrates "how to do it," but cannot perceive or understand "what the user actually does" in real time. If the user makes a mistake in a certain step, the AR device often continues to play the next step of instruction, causing the error to be perpetuated and amplified.

[0005] In summary, both static projection and traditional AR guidance share an inherent technical bottleneck: a lack of real-time interactive feedback between digital guidance information and the physical creation entity. The digital world is essentially "shouting" to the physical world, but receiving no real-time, understandable "echo." This architectural flaw prevents devices from actively calibrating for dynamic deviations during the creation process, and further hinders their ability to provide responsive feedback based on the real-time state of the physical creation. This not only limits the depth of application of digital assistance technologies but also forces operators to expend significant effort on self-checking and correction, making it difficult to improve the efficiency of physical creation. Summary of the Invention

[0006] This application provides a method, device, product, and medium for digital technology-assisted physical creation, which improves the efficiency of creators in physical creation.

[0007] In a first aspect, this application provides a method for digital technology-assisted physical creation, applied to an augmented reality device, the method comprising: In response to the user's instructions for assisting in creation, images of the current physical creation entity are acquired, with the images acquired from at least two perspectives; Perform 3D reconstruction on the current physical creation entity to generate current state data representing the current physical creation entity; The current state data is compared with the preset digital target model to generate deviation data to characterize the difference between the two. Based on the deviation data, calibration visual instructions are generated through preset calibration visual rules; Based on the calibration visual instructions, a calibration mark is displayed on the front-end display interface; this calibration mark is determined by the deviation data. Collect user interaction data in response to calibration vision commands. This user interaction data includes confirmation commands and denial commands. Confirmation commands are used to retain the calibration vision command, and denial commands are used to cancel the calibration vision command.

[0008] In the above embodiments, by acquiring images of the physical entity being created and constructing its digital 3D model, the augmented reality device can perceive the physical entity being created by the user and translate it into a state that it can understand, thereby obtaining the entity's state data. This state data is compared with a preset digital target model to identify differences, which are then visualized to guide the user's creation, enabling proactive calibration of dynamic deviations that occur during the user's creation process. Simultaneously, user interaction provides responsive feedback on the real-time state of the physical creation, reducing the user's effort in self-checking and correction, and improving the user's creative efficiency.

[0009] In conjunction with some embodiments of the first aspect, in some embodiments, a three-dimensional reconstruction of the current physical creation entity is performed to generate current state data characterizing the current physical creation entity, specifically including: The depth point cloud of the physically created entity is obtained through a depth sensor; The image is processed using a neural network to extract and match feature points between each image; Based on the feature points between each image, determine the spatial transformation relationship between depth point clouds; By applying this spatial transformation relationship, the depth point cloud is aligned to the same coordinate system; Combine the aligned depth point clouds into a 3D model; The 3D model is denoised to generate current state data; the current state data includes at least one of the following: 3D point cloud data of the physical creation entity, mesh model, or parametric geometric model.

[0010] In the above embodiments, by combining the 3D depth point cloud acquired by the depth sensor with the visual features extracted from the image by the neural network, on the one hand, spatial anchor points are provided for textureless areas in the depth point cloud using visual features, thereby determining the spatial transformation relationship between different depth point clouds; on the other hand, the overall shape of the physical entity is preserved based on the depth point cloud data. Based on the above spatial transformation relationship, all depth point clouds are unified into the same 3D coordinate space, thereby constructing a machine-readable digital 3D model of the current physical creation entity. The digital state data of the current physical entity is extracted from this model as the original comparison basis for subsequent difference comparison, thereby obtaining quantifiable comparison difference results and improving the user's creative efficiency.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, before comparing the current state data with a preset digital target model to generate deviation data for characterizing the difference between the two, the method further includes: Receive the user's selected target art style instruction; Based on the target art style instruction, a preset stylization processing model is invoked to transform at least one art attribute contained in the preset digital target model, thereby generating a stylized digital target model. The art attribute includes at least one of the model's virtual brushstroke texture, color harmony relationship, and lighting rendering effect. Update the preset digital target model to this stylized digital target model.

[0012] In the above embodiments, through a pre-set stylization processing model, the device can respond to a user's specific artistic style requirements and digitize these requirements, transforming abstract artistic styles into concrete and quantifiable artistic attributes. This allows for a more accurate understanding of the user's creative intent, thereby generating more suitable auxiliary information and improving the user's creative efficiency.

[0013] In conjunction with some embodiments of the first aspect, in some embodiments, the current state data is compared with a preset digital target model to generate deviation data to characterize the difference between the two, specifically including: The current state data and the digital target model are placed in a three-dimensional coordinate system for spatial attitude alignment to obtain the geometric registration result. Based on the geometric registration result, the geometric structure difference between the current state data and the digital target model at the corresponding spatial location is calculated, generating geometric structure deviation data; and / or, Extract and compare the color distribution and / or material features of the current state data with those of the digital target model to generate surface appearance deviation data; This deviation data includes geometric deviation data and / or surface appearance deviation data.

[0014] In the above embodiments, by decomposing the deviation data into two different dimensions—"geometric structure" and "surface appearance"—and calculating the deviation data for each dimension separately, analysis of one or more dimensions of creative deviation is achieved. This design allows the device to provide users with more targeted calibration instructions. For example, when dealing with form deviations in sculptures, the focus can be on guiding geometric deviations; when dealing with coloring or texture issues in paintings, the focus can be on prompting surface appearance deviations. This, in turn, improves the user's creative efficiency.

[0015] In conjunction with some embodiments of the first aspect, in some embodiments, calibration visual instructions are generated based on deviation data and through preset calibration visual rules, specifically including: The system calls upon a pre-set inspiration database to extract the user's creative preference characteristics. The pre-set inspiration database stores creative data recorded during the user's creative process. Calibration visual instructions are generated through preset calibration visual rules; wherein: based on deviation data, the correction content indicated by the calibration visual instructions is determined; and based on creative preference characteristics, the visual presentation form and guidance strategy of the calibration visual instructions are adjusted.

[0016] In the above embodiments, calibration visual instructions are generated in terms of both content and form through preset calibration visual rules. In particular, in terms of form, an inspiration database is preset based on the user's historical operations. When generating calibration visual instructions, the user's creative preference features are extracted from the database. Then, based on the user's creative preference features, the visual presentation form and guidance strategy of the calibration visual instructions are adjusted to a way that is more in line with the user's habits, making the user's creative process more convenient and improving the user's creative efficiency.

[0017] In conjunction with some embodiments of the first aspect, in some embodiments, the calibration visual instruction displays calibration marks on the front-end display interface; the calibration marks are determined by deviation data, specifically including: Receives the user-selected light source type and / or light angle as preview parameters; Add preview parameters to the current status data to obtain new status data; Determine the new deviation data corresponding to the new state data; Based on the calibration visual instructions, a calibration mark is displayed on the front-end display interface; this calibration mark is determined by the new deviation data.

[0018] In the above embodiments, by previewing the creative effect under virtual lighting and generating calibration marks, a visual tool is provided for users' specific lighting creation needs. Simultaneously, by simulating a specific display environment during the creation process, users can potentially discover deviations that are not obvious under the current lighting but would affect the effect under the target lighting. This allows the device to generate more accurate and multi-dimensional calibration visual instructions, resulting in better calibration mark presentation and further improving the user's creative efficiency.

[0019] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: In response to the user's synchronization command, the calibration vision command is synchronized to the second user; Based on the calibration visual instructions, the calibration mark is displayed on the second user front-end display interface.

[0020] In the above embodiments, by synchronizing the calibration visual instructions to a second user other than the current user and simultaneously displaying the calibration mark on the second user's device display interface, the coordination of physical creation guidance information on different devices is realized, supporting multiple users to perform the same physical creation, and also improving the creation efficiency when multiple users perform the same physical creation together.

[0021] In a second aspect, embodiments of this application provide an augmented reality device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the augmented reality device to perform the method described in the first aspect and any possible implementation thereof.

[0022] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an augmented reality device, cause the augmented reality device to perform the method described in the first aspect and any possible implementation thereof.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an augmented reality device, cause the augmented reality device to perform the method described in the first aspect and any possible implementation thereof.

[0024] Understandably, the augmented reality device provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0025] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. This application provides a technical solution that differs from the one-way output of guidance information in existing technologies by collecting the state of physical entities in real time, comparing them with digital targets, and generating dynamic calibration instructions. This solution constructs an information feedback path between digital guidance and physical creation. This two-way information interaction enables the device to provide corresponding auxiliary information based on the real-time progress of creation, thereby improving the user's creative efficiency.

[0026] 2. This application provides technical means for identifying and guiding creative deviations. It can continuously monitor the differences between the physical entity and the target model, and present the deviation information to the user in a visual manner to assist the user in making adjustments during the creative process. This reduces rework caused by the accumulation of deviations, thereby saving creation time and materials, and improving the degree of conformity between the final physical entity and the preset target, thus improving the user's creative efficiency.

[0027] 3. The technical solution of this application can not only be applied to geometric shape calibration, but also adapt to more diverse creative needs and workflows by introducing functions such as artistic style transformation, user preference feature adjustment, virtual lighting preview, and multi-user collaboration. In particular, by introducing a user interaction confirmation mechanism, this solution respects and retains the creator's final decision-making power, realizing human-computer collaboration in creative intent. This improves the efficiency of users creating physical works using digital technology in complex scenarios. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a method for digital technology-assisted physical creation in an embodiment of this application; Figure 2 This is a flowchart illustrating a process for realizing three-dimensional reconstruction of a physically created entity and extracting current state data in an embodiment of this application. Figure 3 This is a schematic diagram of the physical device structure of an augmented reality device in the embodiments of this application. Detailed Implementation

[0029] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0030] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0031] To facilitate understanding, the application scenarios of the embodiments of this application are described below.

[0032] The embodiments of this application can be applied to multiple fields where digital models are used to assist in the creation of physical entities, such as artistic creation (e.g., sculpture, ceramics), industrial design (e.g., car clay models, product prototype verification), vocational education (e.g., arts and crafts teaching), and medical rehabilitation (e.g., prosthetic limb making). Taking a user sculpting clay as an example, the user wears an augmented reality (AR) device (hereinafter referred to as the device) and hopes to use a preset digital 3D model as a reference to shape the clay in their hands, ultimately ensuring that the physical entity is consistent with the digital model in terms of shape, size, etc.

[0033] In related technologies, digital assistance for physical creation can be achieved by employing projection guidance or split-screen reference. The following describes a scenario using projection guidance: The creator places a piece of clay on a workbench, and a projector above projects the frontal outline of the target model onto the clay. The creator sculpts based on this static outline. When sculpting the side, the clay is manually rotated, and the projector is switched to project the side outline. During this process, the projected outline is fixed; the system cannot perceive the actual shape of the clay, nor can it provide feedback on the depth or precision of the sculpting.

[0034] The method employed in this application's embodiments, by real-time sensing of the physical entity's form, comparing it with a digital target, and generating dynamic visual calibration instructions within the user's field of vision, achieves information interaction between the digital system and physical creation. This not only provides guidance but also updates the guidance information in real-time based on the user's actual actions. The following describes a scenario using the method of this application: A user wears AR glasses and sculpts clay. The AR glasses' camera captures images of the clay in real-time and performs 3D reconstruction. The system compares the reconstructed current clay form with the target digital model. If the system detects that an area of ​​the clay is higher than the target model, a semi-transparent red mark is superimposed on the surface of that area in the user's AR field of vision, indicating that this part needs to be cut; conversely, if an area is lower than the target, a blue mark is displayed, indicating that material needs to be added. When the user performs cutting or adding operations, these red and blue marks shrink, fade, and eventually disappear in real-time, providing dynamic and visual guidance for each step of the user's operation. If the user does not agree with the guidance provided by the AR glasses or has their own creative ideas, the user can choose to refuse, and the AR glasses adjust their next action based on the user's feedback.

[0035] As can be seen, the method in this application embodiment can not only provide digital assistance to the physical creation process, but also effectively solve the problems of one-way and static guidance information and lack of real-time perception of the physical world in related technologies, thereby realizing real-time interactive feedback between humans and machines in physical creation tasks.

[0036] To facilitate understanding, the method provided in this implementation will be described in detail below, using the above scenario as an example. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a method for digital technology-assisted physical creation in an embodiment of this application.

[0037] S101. In response to the user's instruction for assisted creation, acquire an image of the current physical creation entity; wherein the image is acquired from at least two perspectives; Among them, the assisted creation requirement command refers to a specific command initiated by the user to start or trigger the device to enter the assisted creation mode. This command can be a preset voice command, gesture action, or signal input through the physical controller, which is not limited here; the current physical creation entity refers to the objective entity that the user is creating, modifying or improving in the physical world.

[0038] It should be noted that the physical creative entity can be an entity with a macroscopic three-dimensional structure, such as sculptures, ceramics, and models; or it can be an entity that has a macroscopic two-dimensional planar form, but whose creative medium has microscopic three-dimensional physical characteristics on its surface, such as oil paintings, prints, and calligraphy works.

[0039] Specifically, this step is performed in the initial stages when a user wants digital model guidance to assist in the creation process, or when the user wants to check the current progress during the creation process.

[0040] When an augmented reality device receives a user's instruction to assist in creation, it invokes one or more built-in image acquisition units (such as RGB cameras or spatial awareness cameras) to capture images of the scene containing the physical creation entity from at least two different perspectives. During this process, the device not only records pixel data, but also uses a built-in spatial positioning module (such as a Simultaneous Localization and Mapping (SLAM) module) to calculate and record the device's own pose information in space (i.e., 3D coordinate position and 3D rotational attitude) when each frame of the image is captured.

[0041] This process ensures that all acquired images can be placed in a unified three-dimensional coordinate system, providing original data input with spatial relationships for three-dimensional reconstruction in S102.

[0042] In some embodiments, optionally, this step can be implemented in the following way: After receiving the user's instruction to assist in creation, the device first uses a virtual viewfinder on the front display interface to lock the physical creation entity and complete the first image acquisition. Then, according to the preset reconstruction algorithm requirements, the device will generate a guiding visual prompt (such as an arrow or target location box) on the display interface to prompt the user to move the device to the next acquisition perspective. When the device determines that it has reached or is close to the viewpoint through its own spatial positioning module (such as SLAM real-time localization and mapping module), it automatically completes the second and subsequent image acquisitions, so that the acquired image data have different parallax relationships, which facilitates subsequent 3D reconstruction.

[0043] Alternatively, a more automated way to accomplish this step is: When the user switches the device to "Assisted Creation Mode," the device activates its image acquisition unit and spatial positioning module to begin image acquisition. The device performs real-time object recognition on the acquired video stream, continuously tracking the position of the physical creation entity while simultaneously recording the device's motion trajectory. When the device detects that its positional movement relative to the entity has accumulated sufficient parallax (e.g., translation exceeding 10 cm or rotation exceeding 15 degrees), it automatically captures and stores a set of high-quality keyframe images and their corresponding pose data from the video stream. Notably, the user does not need to perform any additional interactive operations during this process, achieving seamless and continuous background data acquisition.

[0044] It is understandable that this step can be achieved in other ways, and no specific method is specified here.

[0045] During the implementation of this solution, a potential problem is that during image acquisition, the user's hand or creative tool may partially occlude the physical creative entity, resulting in incomplete image information and affecting subsequent 3D reconstruction. To address this issue, this embodiment provides an occlusion avoidance method based on real-time image segmentation: before the device acquires image data and uses it for reconstruction, it first passes through a preprocessing module. This module incorporates a lightweight semantic segmentation neural network model, which, after training, can identify and segment pixels in the image into different categories such as "creative entity," "hand / tool," and "background" in real time. The device calculates the overlap rate between pixel regions identified as "hand / tool" and pixel regions identified as "creative entity" in real time. When this overlap rate is lower than a preset threshold (e.g., 5%), the frame image is considered "clean" and valid data, and is then transmitted to the subsequent S102 step for processing. If the overlap rate exceeds the threshold, the device will temporarily discard the frame image and simultaneously issue a prompt to the user on the front-end display interface (for example, highlighting the outline of the occluded hand in red), reminding the user to temporarily remove the obstruction, thereby capturing a complete, unobstructed image of the physical creation entity.

[0046] S102. Perform three-dimensional reconstruction on the current physical creation entity to generate current state data representing the current physical creation entity; In this context, 3D reconstruction refers to a technical process used to calculate and generate a digital model representing the geometric shape of a physical creation entity in three-dimensional space from a series of two-dimensional images or other sensor data acquired by the device in S101. Current state data refers to the direct output of this 3D reconstruction process; it is a data file that digitally records the complete or partial spatial morphology of the physical creation entity at the moment of acquisition, for subsequent digital analysis and comparison. This current state data can exist in one or more formats, such as 3D point cloud data, mesh models, or parametric geometric models. 3D point cloud data refers to a collection of numerous vertices, each containing its (X, Y, Z) coordinates in a three-dimensional Cartesian coordinate system, collectively describing the geometric shape of the entity's surface. A mesh model, based on 3D point cloud data, connects adjacent vertices through a specific topological structure, forming numerous polygonal patches (usually triangles or quadrilaterals) to constitute a continuous entity surface. A parametric geometric model, on the other hand, uses mathematical equations (such as NURBS curves and surfaces, basic geometric shapes, etc.) to describe the shape of the entity, representing a higher level of abstraction.

[0047] Specifically, this step is executed after the device completes the multi-view image acquisition in S101, serving as a bridge connecting the perception of the physical world and the analysis of the digital world. The core task of the entire process is to deduce a unique and unified three-dimensional object shape from seemingly isolated two-dimensional photographs taken from different angles.

[0048] First, the device utilizes its internal computing unit to extract features from multiple image data sets obtained by S101 and their corresponding pose information. Second, the device analyzes the parallax between images—the difference in pixel position of the same object point in images from different viewpoints—using an image depth data extraction unit (e.g., a depth sensor) to calculate the spatial depth of that point. By performing this calculation on thousands of points on the surface of the physical object, a preliminary, discrete point cloud is constructed. Subsequently, the device uses the feature information extracted from S101 to perform coordinate transformation and alignment on the local point clouds generated from different viewpoints, ultimately fusing them into a complete overall model located in a unified coordinate system.

[0049] It should be noted that the focus of this step varies depending on the type of physical artwork: when the artwork is a sculpture, 3D reconstruction aims to capture its complete macroscopic outline and volume; while when the artwork is a two-dimensional work such as an oil painting, 3D reconstruction focuses on capturing its surface microscopic topology, namely the brushstrokes and textures formed by the accumulation of paint. In this case, the depth information mainly represents the height distribution of the creative medium on the substrate. The final generated current state data provides an objective and quantifiable data basis for the comparison with the digital target model in subsequent step S103.

[0050] In some embodiments, this step can be implemented in a variety of ways: Alternatively, this can be achieved through a method based on the fusion of depth sensors and visual features.

[0051] For easier understanding, please refer to Figure 2 This is a flowchart illustrating a process for realizing the three-dimensional reconstruction of a physical creation entity and extracting current state data in this embodiment.

[0052] In this embodiment, firstly, based on the image viewpoints determined in S101, depth point clouds are acquired using depth sensors at these viewpoints.

[0053] Subsequently, the internal neural network processor processes the images acquired in S101 corresponding to the aforementioned viewpoints. It extracts stable corner, edge, and texture feature points from each image using a convolutional neural network and matches them across different images to find corresponding feature points in the same physical space. Based on the successfully matched feature point pairs, one or more spatial transformation matrices are calculated. These matrices contain the mathematical relationships required to rotate and translate the coordinate system of one viewpoint to that of another. The device then applies this transformation relationship to the raw depth point clouds directly acquired by the depth sensor, which are scattered across their respective local coordinate systems. It aligns and registers all these points to the same global coordinate system and merges them into a unified, denser point cloud.

[0054] Finally, the device performs denoising and smoothing algorithms on the fused point cloud to remove outliers caused by sensor errors or environmental interference. It can also selectively convert the point cloud into a mesh model with a continuous surface using surface reconstruction algorithms (such as Poisson surface reconstruction), generating the final usable current state data. For example, when scanning an oil painting with impasto techniques, the device's depth sensor will operate in a high-resolution mode to capture micrometer-level height variations caused by paint thickness, thereby generating current state data that reflects the surface texture.

[0055] Alternatively, S102 can be implemented using another path: The device first uses only the image sequence acquired in S101 to analyze the feature point matching relationship between the images through the SFM algorithm, so as to simultaneously calculate the relative pose of all acquisition views and a sparse scene 3D point cloud. Then, the device uses the calculated pose information to perform pixel-level dense matching through the MVS algorithm to perform pixel-by-pixel depth estimation of the scene, thereby generating a very dense point cloud. Finally, after post-processing steps such as point cloud denoising, fusion and meshing, the final current state data is generated.

[0056] It is understandable that other methods can be used to achieve this 3D reconstruction process, such as methods that directly acquire high-density point clouds based on laser scanning or time-of-flight (ToF) cameras, which are not limited here.

[0057] A key issue that may arise during the implementation of this solution is the lack of sufficient texture features or the presence of optical properties such as high reflectivity or transparency on the surface of the physical creation entity. Both of these situations severely interfere with the extraction and matching of feature points or the perception of depth during the 3D reconstruction process, leading to large areas of voids, deformation, or complete failure in the reconstructed model. To address this issue, this embodiment provides an adaptive reconstruction strategy based on active structured light projection: When the device performs the reconstruction task in S102, it incorporates a model quality assessment module. This module analyzes the integrity of the preliminary reconstruction results in real time, for example, calculating the effective point cloud coverage or detecting the area of ​​damaged regions in the model. When the assessment module determines that the current reconstruction quality is below a preset threshold (e.g., the void area of ​​the model exceeds 10% of the total surface area), the device identifies a "difficult surface" and automatically switches to active scanning mode. In this mode, the device activates its built-in micro-projection unit (such as a Pico Projector) to project a preset random speckle pattern or coded grating pattern with high-frequency information onto the surface of the physical creation entity. This projected pattern artificially creates rich and stable visual features on the originally smooth or highly reflective surface. The device then prompts the user to repeat the image acquisition step in S101, ensuring that the acquired image includes the projected pattern. The device then uses these patterned images to perform the 3D reconstruction process in S102. Due to the presence of the artificial texture, the success rate and density of feature point matching are significantly improved, enabling the generation of a complete and continuous geometric model and reducing reconstruction failures caused by object surface characteristics. For example, when scanning a pure white plaster prototype or a ceramic blank with a transparent glaze, this adaptive strategy ensures that the device consistently outputs usable current state data.

[0058] In some preferred embodiments, in order to provide richer artistic guidance during the creative process, rather than simply replicating geometric shapes, the method may optionally include the following steps before performing step S103: Receive the user's selected target art style instruction; Based on the target art style instruction, a preset stylization processing model is invoked to transform at least one art attribute contained in the preset digital target model, thereby generating a stylized digital target model. The art attribute includes at least one of the model's virtual brushstroke texture, color harmony relationship, and light and shadow rendering effect. Update the preset digital target model to the stylized digital target model.

[0059] The target art style instruction refers to a control signal input by the user through the device's human-computer interaction interface, used to specify a desired art style or a set of art parameters.

[0060] A stylized processing model refers to a pre-built computational model stored on a device that encapsulates the algorithmic logic for applying specific artistic style features to a general digital model, such as a trained deep neural network. Artistic attributes refer to the quantifiable and modifiable multi-dimensional features that constitute the visual representation of a digital model; these features collectively define the model's artistic appeal. Specifically, virtual brushstroke textures represent the geometric details generated on the surface of a digital model through normal mapping, displacement mapping, or direct modification of the model's surface mesh, used to simulate tool marks (such as brushstrokes and carving marks) in real painting or sculpture. Color harmony relationships represent the overall hue, saturation, brightness distribution, and color matching rules of the model's surface texture map. Lighting and shadow rendering effects refer to the model's own material parameters, such as specular reflectance, diffuse reflectance, and surface roughness. These parameters determine how the model reflects and absorbs light in a virtual lighting environment, thus presenting specific visual effects.

[0061] Stylized digital target models refer to new versions of model data generated after the original digital target model has been stylized and transformed according to artistic attributes.

[0062] Specifically, this series of steps is typically triggered after the user loads the initial digital target model and before they begin assisting the user with physical creation. The goal is to transform a standard geometric model, which may not have obvious artistic features, into a creative blueprint that embodies a specific artistic style (such as Picasso's style), making subsequent creative assistance more artistically instructive.

[0063] When the process starts, the device presents the user with a style library or a series of parameter adjustment tools. The user selects a preset style (such as "Classicism" or "Expressionism") or manually adjusts several artistic attribute parameters according to their creative intent, thereby generating a target artistic style instruction. Upon receiving this instruction, the device retrieves and activates the corresponding stylization processing model. This model takes the original preset digital target model as input and calculates and transforms one or more of its artistic attributes based on the style defined in the instruction. For example, for a smooth digital portrait model, after applying the "Rodin style" instruction, the stylization processing model might add numerous virtual brushstroke textures simulating hand-sculpted clay marks to its surface and adjust its lighting and shadow rendering effects to emphasize the volume of muscle groups. After processing, the generated stylized digital target model will replace the original preset digital target model in the device's memory or storage. The subsequent S103 comparison step will use this new model, "infused with artistic soul," as the comparison benchmark.

[0064] In some embodiments, the data processing logic of this stylization process is as follows: The user selects the "Impressionism" option from the style list on the device's virtual display, and the device then receives the instruction for that target art style.

[0065] Based on this instruction, the control unit invokes the stylized processing model associated with "Impressionism," which contains two sub-modules: a brushstroke generation module and a color mapping module. The device loads the data of the original digital target model (such as a 3D model of a landscape) with a smooth surface into the processing flow.

[0066] First, a normal map is generated on the surface of the original digital target model using the brushstroke generation module. The distribution of normal vectors in this map simulates the short, juxtaposed brushstrokes in Impressionist paintings. This normal map is then added as an artistic attribute of the model.

[0067] Next, the color mapping module reads the model's original color texture map and, based on preset Impressionist color theory (e.g., emphasizing the separation of light and color), performs color space transformation on the pixels of the entire texture map, generating a new color map with higher color saturation and more vivid contrast. This new map updates the model's color harmony. Finally, the device integrates and packages the model with the new normal map and color map, generating a stylized digital target model, which is then set as the target for the current creative task. Subsequent comparisons and guidance will revolve around helping users reproduce this "Impressionist" effect with specific brushstrokes and color relationships on a physical entity.

[0068] S103. Compare the current state data with the preset digital target model to generate deviation data to characterize the difference between the two. The preset digital target model refers to a pre-defined digital 3D model (e.g., a sculpted human figure model with standard sculpture proportions) provided to the user for creative reference before the creation process begins. It serves as a reference and blueprint for the final form throughout the creative process. It should be noted that the preset digital target model here can be the original model determined before S102, or a stylized digital target model generated after stylization processing, as described in some embodiments. Deviation data refers to the structured output result generated by the comparison process. It is not a simple judgment of "match" or "dismatch," but a detailed dataset used to describe the specific differences between the current state data and the digital target model in terms of space and attributes. For example, deviation data can be a pseudo-color image overlaid on the model surface, where different colors represent the material thickness that needs to be increased or decreased at a certain point on the physical entity's surface.

[0069] Specifically, this step is executed immediately after the device generates the current state data of the physical creation entity through S102, and may be triggered periodically and repeatedly throughout the creation process. It constitutes the core computational link for realizing the creation assistance function of this invention, and its task is to answer the key question: "What, where, and to what extent are there differences between the current physical creation progress and the final digital creation goal?"

[0070] The device utilizes its internal computing units to load the current state data (e.g., a point cloud) generated by S102 and the preset digital target model (e.g., a mesh model) into the same virtual 3D space. The device first aligns the two spatially to ensure the comparison is performed on a unified coordinate system. After alignment, the device performs difference calculations point-by-point or region-by-region. This calculation can be geometric, such as measuring the spatial distance from a point on the physical entity's surface to the corresponding point on the digital target model; or it can be based on appearance attributes, such as comparing the color, texture, or material parameters of the two at the same location. All these calculated difference values ​​are organized into a structured deviation data set and stored. This data directly reflects the inconsistency between the physical creation entity and the digital blueprint, serving as the direct basis for S104 to generate user feedback information.

[0071] In some embodiments, S103 can be implemented in the following manner: The current state data and the digital target model are placed in a three-dimensional coordinate system for spatial orientation alignment to obtain geometric registration results. Based on the geometric registration results, the geometric structural differences between the current state data and the digital target model at corresponding spatial locations are calculated to generate geometric structural deviation data; and / or, Extract and compare the current state data with the color distribution and / or material features of the digital target model to generate surface appearance deviation data; The deviation data includes the geometric deviation data and / or the surface appearance deviation data.

[0072] The device first performs spatial attitude alignment, placing the current state data (such as point cloud) generated by S102 and the digital target model (such as mesh) into the same three-dimensional coordinate system. The device uses an iterative nearest-point algorithm to find a rotation and translation transformation that minimizes the overall distance between the point set of the current state data and the surface of the digital target model through repeated calculations, thereby obtaining a geometric registration result and ensuring that the attitudes of the two are matched and coincident in space.

[0073] Based on this, the device performs one or two difference calculations: On the one hand, the device calculates geometric deviation data by traversing every 3D point in the current state data and calculating the shortest directed distance from that point to the registered digital target model surface. If the point is outside the model surface, the distance is positive, indicating material redundancy; if it is inside the model surface, the distance is negative, indicating material deficiency. The set of distance values ​​for all points constitutes the geometric deviation data. On the other hand, if both the current state data and the digital target model contain color or texture information, the device projects the points in the current state data onto the surface of the digital target model, extracts the color values ​​(e.g., RGB values) at corresponding locations, and then calculates the differences between these color values ​​(e.g., Euclidean distance in the CIELAB color space). The set of all difference values ​​constitutes the surface appearance deviation data.

[0074] Ultimately, the generated deviation data can include geometric deviation data, surface appearance deviation data, or a combination of both. For example, when assisting a user in creating a clay head sculpture, the deviation data mainly reflects geometric deviations (such as the nose being 5 millimeters shorter than expected); while when assisting a user in copying an oil painting, the deviation data may simultaneously include geometric deviations (such as insufficient thickness of paint layer in a certain area) and surface appearance deviations (such as the mixed color being darker than the target color in that area).

[0075] It is understandable that other methods can be used to achieve this comparison process, such as comparison based on deeper geometric features such as model curvature and topological structure, which are not limited here.

[0076] A key issue that may arise during the implementation of this solution is that, during the long-term, multi-stage creation process, the position or orientation of the physical creation entity may unintentionally change (e.g., the user moves the sculpture base), or the accumulated minor errors from multiple scans in S102 may cause drift or misalignment in the geometric registration results calculated in S103, leading to incorrect deviation data and misleading the user. To address this issue, this embodiment provides a global pose correction method based on reference markers: Before creation begins, the user places several (e.g., three or more) physical markers with unique visual characteristics on the workbench of the physical creation entity or on one of its fixed bases. Simultaneously, corresponding virtual markers are also set at their spatial locations within the pre-defined virtual scene of the digital target model. Before each comparison process in S103, the device first performs a global pose pre-correction step. In this step, the device uses an image sensor to identify the current spatial positions of all physical markers in the field of view and calculates their coordinates relative to the device itself. Subsequently, the device solves a rigid body transformation (i.e., the PnP problem) to find a spatial transformation matrix that aligns the currently observed array of physical markers with the array of virtual markers. This matrix represents the true pose of the physical creation entity in the global coordinate system at the current moment. The device uses this high-confidence transformation matrix as the initial value for subsequent registration algorithms such as ICP. In this way, even if the physical entity is moved or rotated, the device can always "pull" it back to the correct global coordinate system before making a fine comparison, thereby avoiding the problems of posture drift and accumulated error, and improving the consistency and reliability of deviation data throughout the entire creation cycle.

[0077] S104. Based on the deviation data, generate calibration visual instructions through preset calibration visual rules; The preset calibration visual rules refer to a set of conditional judgments and mapping algorithms embedded in the device program. These rules define how to convert one or more types of deviation data (such as geometric deviation and color deviation) into one or more specific visual representations. The calibration visual instruction is the final output of this conversion process. It is a structured rendering command that the device can directly execute to drive the display or projection unit, generating guiding visual signals that can be intuitively understood by the human eye. For example, a calibration visual instruction might contain the following information: "Render a hemisphere with radius r, color #FF0000, and transparency of 50% at position (x, y, z) in the world coordinate system." This instruction is generated based on the information in the deviation data that "material is missing by 5 millimeters at a certain location" and according to the calibration visual rule that "negative geometric deviation is mapped to a red semi-transparent body."

[0078] Specifically, this step is triggered after the device completes the difference comparison of S103 and generates deviation data. It transforms the numerical deviation data output by S103, which is machine-readable but not intuitive for users, into a user-friendly and clearly perceptible visual guidance scheme.

[0079] The device retrieves its internally stored calibration visual rule library and iterates through the deviation data generated by S103. For each difference point or region in the deviation data, the device matches and applies corresponding visual rules based on its attributes (geometric structure deviation or surface appearance deviation) and magnitude (how large the difference). For example, for areas where material needs to be added (negative geometric deviation), a warm-toned (e.g., red, orange) virtual block is generated and superimposed on the corresponding physical entity location; for areas where material needs to be removed (positive geometric deviation), a cool-toned (e.g., blue, green) virtual block is generated. The magnitude of the deviation can also be reflected by the saturation, brightness, or flicker frequency of visual elements; the greater the magnitude, the stronger the visual cues. For color deviation, the rule could be to display a directional arrow in the area with color difference, and side-by-side display of color blocks for the current color and the target color for comparison. Through this series of processes, a complex deviation data report is transformed into a complete, multi-layered calibration visual instruction set that will be presented to the user.

[0080] In some embodiments, S104 can be implemented in the following manner: Before generating instructions, the device first accesses a pre-set inspiration database. This database stores a large amount of creative data recorded during the user's past creative process, such as the user's operating habits, response time to different types of guidance information, and the extent of modifications. Through this historical data, the device can extract the user's creative preference characteristics, such as determining whether the user tends to make sweeping changes or make meticulous fine-tuning; whether they are more likely to understand color cues or are more sensitive to geometric guidance.

[0081] When generating calibration visual instructions, the device executes a two-track process: On one hand, it still uses the deviation data from S103 and basic calibration visual rules to determine the core correction content to be indicated by the instruction. On the other hand, the device uses the extracted creative preference features to dynamically adjust the final visual presentation and guidance strategy of these instructions. For example, for a creative preference feature judged as "beginner," when the device needs to indicate a correction for "missing material," the generated calibration visual instruction might be a highly saturated (saturation of 60% or higher) red solid overlay model with a flashing effect; while for an "expert" feature, the same correction content might only be presented as a low-saturation (saturation of less than 60%) semi-transparent red outline to avoid excessively interfering with the expert's creative intuition. Ultimately, this calibration visual instruction, which integrates objective correction content and personalized presentation, is generated and output.

[0082] It is understandable that other methods can also be used to achieve this, such as introducing dynamic adjustment mechanisms based on physiological feedback (e.g., eye tracking), which are not limited here.

[0083] A core issue that may be encountered during the implementation of this solution is "guidance failure" or "visual saturation." When the difference between the physical creation entity and the digital target model is too large, S103 will generate a massive amount of deviation data. If S104 indiscriminately converts all deviations into visual instructions, the screen or projection image will be filled with a large number of overlapping and cluttered visual cues (such as a screen full of red and blue patches), making it impossible for the user to distinguish between primary and secondary elements and find the focus of the operation, thus making the guidance information itself a distraction. To solve this problem, this embodiment provides an instruction generation strategy based on deviation data grading and dynamic focusing: After receiving the deviation data from S103, the device will not immediately perform global rule mapping, but will first execute a preprocessing step. This step first divides all deviation points into multiple priorities based on the absolute size of the deviation value, such as "severe deviation" (e.g., deviation greater than 10mm), "moderate deviation" (2mm to 10mm), and "minor deviation" (less than 2mm). Then, the device will generate calibration visual instructions only for the "severe deviation" level data by default and present these instructions to the user. Simultaneously, the device provides an interactive mechanism, such as a physical button or virtual slider, allowing users to switch between guidance information of different priorities. Once the user has addressed all severely flawed areas, they can actively switch to the "moderate flaw" guidance view. At this point, the device hides the corrected area hints and generates and displays visual instructions for the moderate flawed areas. Through this hierarchical display and step-by-step guidance, the device breaks down a complex and challenging correction task into a series of clear and focused sub-tasks, avoiding information overload and providing users with targeted visual guidance at different stages of the creative process.

[0084] S105. Display calibration marks on the front-end display interface according to the calibration visual instructions; these calibration marks are determined by the deviation data. Here, the front-end display interface refers to the physical or virtual screen through which the user interacts with the device, such as the lens of a head-mounted display in augmented reality (AR) applications. The calibration mark, on the other hand, is a specific visual element presented to the user on the front-end display interface after the calibration visual instructions generated in S104 are executed. It visually indicates the location, type, and degree of difference between the physical creation entity and the digital target model. For example, a calibration mark could be a highlighted color block superimposed on the surface of the entity, a set of guiding outlines, a dynamic arrow indicating direction, or a virtual geometry simulating the "addition" or "reduction" of material; no limitation is made here. All attributes of the calibration mark, such as its shape, color, and position, are ultimately determined by the deviation data generated in S103 and the rules in S104.

[0085] Specifically, this step is the final presentation stage of the entire creative assistance information flow. It is executed after generating calibration visual instructions inside the device, and transforms the abstract correction suggestions into visual signals that the user can directly perceive.

[0086] The device drives its graphics processing unit to perform real-time rendering on the front-end display interface according to the instruction set output by S104. In augmented reality applications, the device first captures images of physical creation entities in the real world using a camera, and then uses spatial positioning and tracking technology to "anchor" calibration markers in three dimensions to the corresponding spatial positions of the physical entities. This means that even if the user moves their viewpoint or moves the physical entity, the calibration markers will move accordingly, providing the user with an immersive, WYSIWYG guided experience. For example, if the deviation data shows that the bridge of the nose needs to be raised by 2 millimeters, the device will overlay a red, semi-transparent virtual block representing "raising by 2 millimeters" on the real bridge of the nose in the user's field of vision as a calibration marker, which the user can then refer to for material filling.

[0087] In some embodiments, S105 can be implemented in the following manner: The device first receives the user's selected light source type and / or light angle as preview parameters through the front-end display interface.

[0088] For example, users can select "parallel light" as the light source type in the interactive window to set a light angle that falls from the upper left at a 45-degree angle. After receiving the "parallel light" preview parameter, the device will use this parameter to re-render the current state data acquired by S102 in its internal virtual 3D scene, obtaining new state data that simulates a specific lighting effect. At the same time, the device will also render a preset digital target model under the same lighting conditions.

[0089] Subsequently, the device will re-execute the comparison process to determine the differences in light and shadow performance between the current state data and the target model under the new lighting conditions. For example, some areas may have excessively heavy shadows or insufficient highlights. These differences constitute new bias data.

[0090] Finally, the device generates and displays calibration marks based on this new deviation data, according to the calibration visual rules of S104. In this way, the displayed calibration marks no longer merely reflect differences in geometric shapes, but also indicate deficiencies in the lighting and shadow rendering of the artwork under the target lighting environment selected by the user. This makes the user's correction operations more art-oriented, not just "replicating" a model, but "sculpting" light and shadow.

[0091] It is understandable that other methods can also be used to achieve this, such as allowing users to select different virtual materials (such as bronze and marble) as preview parameters to preview and guide the shaping of different material textures; this is not limited here.

[0092] A key issue that may arise during the implementation of this solution is that when calibration markers are displayed on physical entities using augmented reality with an overlaid visual effect, the user's hand, tools, or parts of the physical creation entity itself may obscure the target area to be observed and manipulated, thus also obscuring the calibration markers overlaid on that area, preventing the user from seeing the guidance information. To address this issue, this embodiment provides a display method based on depth perception-based occlusion penetration and information reconstruction: when rendering calibration markers, the device utilizes scene depth information acquired by a depth sensor in real time. When the device detects that the pixel position of the calibration marker is obscured by an object closer to the camera (such as the user's creation tool), it does not simply stop rendering the marker. Instead, the device activates a set of alternative display rules. One rule is to re-render the obscured calibration marker on top of the obscuring object with a special visual style (e.g., a semi-transparent, shimmering dashed outline). In this way, although the user cannot see the details of the entity under the marker, they can clearly perceive the "existence" of the guidance marker and its position and extent in space. Another more advanced approach is to create a "picture-in-picture" window in one corner of the display interface. This window renders a completely virtual, unobstructed physical entity and calibration markers from different virtual perspectives, allowing users to easily refer to an unobstructed view to understand operating instructions. Through this intelligent occlusion handling mechanism, the device ensures the effective transmission of calibration information even in complex operating environments.

[0093] S106. Collect user interaction data of user response to calibration vision command, wherein the user interaction data includes confirmation command and denial command, the confirmation command is used to retain the calibration vision command, and the denial command is used to cancel the calibration vision command.

[0094] User interaction data refers to the user's responsive actions to calibration markers displayed on the front-end interface, which can be recognized and interpreted by the device. A confirmation instruction is a specific type of user interaction data; it indicates that the user acknowledges the guidance of the current calibration visual instruction and intends (or has already) to perform physical creative operations as instructed. Upon receiving this instruction, the device will retain the corresponding calibration marker or mark it as "completed." Conversely, a denial instruction is another type of user interaction data; it indicates that the user disagrees with or temporarily ignores the current calibration visual instruction. For example, if the user believes that the modification does not conform to their creative intent, the device, upon receiving this instruction, will cancel or hide the corresponding calibration marker, thus ceasing to guide the user.

[0095] Specifically, this step continues after S105 displays the calibration mark to the user, and it is a key link in forming an interactive feedback loop between the device and the user. The device is no longer just providing one-way "guidance," but begins to "listen" to the user's intentions, thus making the entire assisted creation process more interactive and flexible.

[0096] The device activates its input monitoring modules (such as touchscreen controllers, spatial locators, and voice recognition engines) to specifically wait for and capture user responses to specific calibration markers. When a user sees a calibration marker, they may make a judgment. If the user agrees with the guidance, they might "confirm" it through a gesture, a click, or a voice command. The device captures this action and generates user interaction data containing the marker ID and the "confirmed" status. If the user disagrees, for example, if they believe the device's suggestion would ruin a subtle detail of their work, they can also "deny" the instruction through a corresponding action, and the device generates data containing a "no" status. This collected user interaction data is recorded and used not only to adjust the displayed content of the front-end interface in real time (keeping or removing the marker) but also as historical data for analyzing user creative preferences and habits.

[0097] It should be noted that if the user's confirmation data is a confirmation command, the device will continue to retain the current calibration visual command, waiting for the user to continue creating on the physical entity according to the calibration mark obtained from the current calibration visual command; if the user's confirmation data is a denial command, the device will cancel the current calibration visual command and will no longer generate calibration marks.

[0098] In some embodiments, S106 can be implemented in a variety of ways: Optionally, this can be achieved through explicit graphical user interface (GUI) interaction. The processing logic is as follows: First, in S105, while rendering each calibration mark, the device also renders two small, interactive virtual buttons nearby, such as a "√" icon and an "×" icon. Next, the device continuously monitors events on its input device (such as a touchscreen or an AR controller's ray). When the device detects that a user's click or hover action hits the "√" icon associated with a calibration mark, it immediately generates a confirmation command. This command data packet contains the unique identifier of the confirmed calibration mark, the operation timestamp, and a confirmation command of type "yes." The device then changes the calibration mark's incomplete state (e.g., red) to a confirmed state (e.g., green) on the graphical user interface. Conversely, if the device detects that the user clicked the "×" icon, it generates a denial command containing the same elements but of type "no," and immediately removes the corresponding calibration mark from the interface.

[0099] Optionally, the device can also implicitly collect user interaction data by analyzing the user's body language or physical operations. First, the device defines corresponding "operation response zones" in its internal 3D spatial model based on the displayed calibration markers. For example, for a calibration marker indicating "material missing," the corresponding operation response zone is the recessed area on the physical entity that needs to be filled. Then, the device performs local 3D scans of these operation response zones at a high frequency (e.g., several times per second), continuously acquiring the current state data of these areas. Finally, the device compares and analyzes the real-time scanned data with the data from the previous moment. If the device detects a continuous, positive increase in the geometric volume within a response zone indicating "material missing," and the increase exceeds a preset threshold (e.g., reaching 70% of the target correction amount), the device determines that the user is performing an operation consistent with the instruction, thus automatically generating a confirmation instruction. Conversely, if the user does not perform any operation within a response zone for an extended period (e.g., exceeding a preset time length), or performs an operation contrary to the instruction (e.g., continuing to remove material), the device can generate a denial instruction.

[0100] It is understandable that other methods can also be used, such as using voice recognition to collect users' verbal commands like "confirm this change" or "skip here," which is not limited here.

[0101] A key issue that may arise during the implementation of this solution is that users may neither explicitly confirm nor deny a calibration visual instruction, but instead choose to temporarily set it aside and work on other parts of their work. If the device simply interprets this "no action" as "denial," it may mistakenly cancel an important prompt that the user intended to handle later, thus contradicting the user's true intention. To address this issue, this embodiment provides an interaction mechanism that introduces a "pending" state and an "instruction storage area": ​​When collecting user interaction data, in addition to the two basic instructions of "confirm" and "deny," the device adds a "pending" state. When a user does not perform an explicit operation on a calibration mark for an extended period, nor an implicit operation on its corresponding physical area, the device will not immediately interpret it as a denial. Instead, it will automatically classify the calibration visual instruction into the "pending" state and fade or hide the corresponding calibration mark from the main viewpoint to reduce interference with the current creative focus. Simultaneously, the device provides a persistent and always-accessible "instruction storage area" icon on the front-end display interface. Clicking the icon will bring up a list or thumbnail view displaying all calibration commands in a "pending" state. In this buffer, users can review, reactivate, or finally confirm / deny their previously shelved modification suggestions. This gives users greater freedom in their command interaction, maintaining the simplicity of the main creation interface while avoiding information loss due to misinterpretations of user silence.

[0102] It should be noted that the assisted creation process of S101-S107 will be continuously executed during the process of the device assisting the user in creating physical entities, until the user makes an action or instruction to cancel the use of the device for assisted creation, such as the user turning off the power of the device.

[0103] In some preferred embodiments, to support advanced application scenarios such as collaborative creation and remote guidance, the functionality of this solution can be further expanded in the following ways: In response to the user's synchronization command, the calibration vision command is synchronized to the second user; According to the calibration visual instructions, the calibration mark is displayed on the second user front-end display interface.

[0104] In this context, the user's synchronization command refers to a specific command initiated by the user currently operating the device (hereinafter referred to as the first user for ease of understanding) with the intention of sharing one or more calibration visual instructions with other participants. Optionally, this command can be triggered by clicking the "Share" button on the interface, performing a preset gesture (such as "grabbing" a calibration mark in the augmented reality environment and "moving" it towards the icon representing the second user), or issuing a voice command such as "Send this prompt to [second user name]", without limitation here.

[0105] In a collaborative creation scenario, the second user refers to a participant other than the first user who initiated the synchronization command. This user also possesses a device capable of executing this solution, and the device is powered on and in use. Synchronization means transmitting a calibration visual command data packet, containing complete rendering and positioning information, generated on the first user's device, to the second user's device via network communication, ensuring that both parties see semantically and spatially consistent guidance information.

[0106] Specifically, this implementation method is mainly applied to scenarios where two or more people collaborate on the same physical creation, such as an experienced instructor guiding a student in creating a sculpture, or two designers jointly modifying a physical model. The implementation process of this embodiment is triggered after the first user's device completes S104 and generates a calibration vision instruction.

[0107] When the first user sees the calibration markers displayed by S105 on their front-end display interface and believes that the modification indicated by a certain marker needs to be communicated to the second user, the first user will issue a synchronization command. After receiving the synchronization command, the first user's device locates the calibration visual command specified by the user and encapsulates the command type (such as "add material"), geometric description (such as the shape and size of the virtual block to be rendered), visual attributes (such as color and transparency), and its three-dimensional coordinates and pose in the shared physical creation entity coordinate system into a data packet, which is then sent to the second user's device through a preset transmission medium (such as a wireless network).

[0108] After receiving the data packet, the second user's device splits the packet and executes the process in step S105. Finally, on the second user's front-end display, a calibration mark identical to that seen by the first user and similarly "anchored" to the same physical entity is displayed. This allows the two users to communicate and collaborate around the same creative point, even if they are in different locations or have different viewing angles, thus providing a broader dimension for real-time interactive feedback in digital-assisted physical creation, building upon the previous embodiment.

[0109] The augmented reality device in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the physical device structure of an augmented reality device in the embodiments of this application.

[0110] It should be noted that, Figure 3 The structure of the augmented reality device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0111] like Figure 3 As shown, the augmented reality device includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or loaded from storage portion 308 into random access memory (RAM) 303, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 303. The CPU 301, ROM 302, and RAM 303 are interconnected via bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0112] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0113] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0114] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0116] Specifically, the augmented reality device in this embodiment includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the method for digital technology-assisted physical creation provided in the above embodiment.

[0117] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the augmented reality device described in the above embodiments; or it may exist independently and not incorporated into the augmented reality device. The storage medium carries one or more computer programs that, when executed by a processor of the augmented reality device, cause the augmented reality device to implement the method for digital technology-assisted physical creation provided in the above embodiments.

[0118] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0119] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for digital technology-assisted physical creation, applied to augmented reality devices, characterized in that, include: In response to the user's instructions for assisted creation, an image of the current physical creation entity is acquired, and the image is acquired from at least two perspectives; Perform three-dimensional reconstruction on the current physical creation entity to generate current state data representing the current physical creation entity; The current state data is compared with a preset digital target model to generate deviation data that characterizes the difference between the two. Based on the deviation data, calibration visual instructions are generated according to preset calibration visual rules; According to the calibration visual instructions, display calibration marks on the front-end display interface; The calibration mark is determined by the deviation data; Collect user interaction data in response to the calibration visual command, wherein the user interaction data includes a confirmation command and a denial command, the confirmation command is used to retain the calibration visual command, and the denial command is used to cancel the calibration visual command.

2. The method according to claim 1, wherein the step of performing three-dimensional reconstruction of the current physical creation entity to generate current state data characterizing the current physical creation entity specifically includes: The depth point cloud of the physical creation entity is obtained using a depth sensor; The images are processed using a neural network to extract and match feature points between each image; Based on the feature points between each image, determine the spatial transformation relationship between the depth point clouds; By applying the aforementioned spatial transformation relationship, the depth point cloud is aligned to the same coordinate system; Combine the aligned depth point clouds into a 3D model; The 3D model is denoised to generate the current state data; the current state data includes at least one of the 3D point cloud data, mesh model, or parametric geometric model of the physical creation entity.

3. The method according to claim 1, before comparing the current state data with a preset digital target model to generate deviation data to characterize the difference between the two, further comprising: Receive the user's selected target art style instruction; Based on the target art style instruction, a preset stylization processing model is invoked to transform at least one art attribute contained in the preset digital target model, thereby generating a stylized digital target model. The art attribute includes at least one of the model's virtual brushstroke texture, color harmony relationship, and light and shadow rendering effect. Update the preset digital target model to the stylized digital target model.

4. The method according to claim 1, wherein comparing the current state data with a preset digital target model to generate deviation data for characterizing the difference between the two specifically includes: The current state data and the digital target model are placed in a three-dimensional coordinate system for spatial orientation alignment to obtain geometric registration results. Based on the geometric registration results, the geometric structural differences between the current state data and the digital target model at corresponding spatial locations are calculated to generate geometric structural deviation data; and / or, Extract and compare the current state data with the color distribution and / or material features of the digital target model to generate surface appearance deviation data; The deviation data includes the geometric deviation data and / or the surface appearance deviation data.

5. The method according to claim 1, wherein generating calibration visual instructions based on the deviation data using preset calibration visual rules specifically includes: The system calls upon a preset inspiration database to extract the user's creative preference features. The preset inspiration database stores creative data recorded during the user's creative process. A calibration visual instruction is generated by means of a preset calibration visual rule; wherein: the correction content indicated by the calibration visual instruction is determined based on the deviation data; and the visual presentation form and guidance strategy of the calibration visual instruction are adjusted based on the creative preference characteristics.

6. The method according to claim 1, characterized in that, According to the calibration visual instructions, calibration marks are displayed on the front-end display interface; the calibration marks are determined by the deviation data and specifically include: Receives the user-selected light source type and / or light angle as preview parameters; Add preview parameters to the current status data to obtain new status data; Determine the new deviation data corresponding to the deviation data under the new state data; According to the calibration visual instructions, calibration marks are displayed on the front-end display interface; the calibration marks are determined by the new deviation data.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to the user's synchronization command, the calibration vision command is synchronized to the second user; According to the calibration visual instructions, the calibration mark is displayed on the second user front-end display interface.

8. An augmented reality device, characterized in that, The augmented reality device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the augmented reality device to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the augmented reality device, the augmented reality device performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on an augmented reality device, the augmented reality device performs the method as described in any one of claims 1-7.

Images