Augmented reality-based virtual-real fusion display method and device for calligraphy and painting

Abstract

This invention discloses a method and device for displaying calligraphy and painting using augmented reality (AR), relating to the field of AR technology. The method includes: scanning the display screen of calligraphy and painting works on an eye-tracking digital terminal and constructing a dense geometric layer and a sparse tracking layer of a two-layer data architecture; extracting information anchor points of specific elements and generating a set of information anchor points; performing eye tracking and gaze point calculation on the user to obtain precise gaze point coordinates, and performing hit determination and activation to obtain the target anchor point hit by the user's gaze; constructing a substrate and potential field environment and activating a multi-target enzyme-based optimization algorithm to solve for the user's three-dimensional pose coordinates; and dynamically overlaying and rendering the target digital information data package onto the display screen of the calligraphy and painting works based on the three-dimensional pose coordinates, achieving a fusion display of calligraphy and painting. This invention solves the problems of AR tag occlusion, rigid layout, cumbersome interaction, and low rendering efficiency in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and in particular to a method and apparatus for displaying calligraphy and painting using augmented reality in a virtual-real fusion manner. Background Technology

[0002] With the development of digital museums and digital technologies, Augmented Reality (AR) technology is being increasingly widely used in cultural exhibition venues such as museums and art galleries. Traditional methods of appreciating calligraphy and paintings are evolving towards digitalization and interactivity. Through AR technology, digital information such as the historical background, artistic techniques, and seal inscriptions behind calligraphy and paintings can be overlaid onto physical exhibits, greatly enriching the viewing experience. Most existing AR calligraphy and painting display applications are based on simple planar recognition technology, triggering the overlay of fixed digital content by scanning QR codes or specific patterns on the scroll.

[0003] However, existing technologies have the following significant drawbacks in practical applications: 1) Obstruction and rigid layout: Traditional AR tags are usually fixed in a specific position, which can easily obscure the core brushstroke details of calligraphy and painting works and destroy the overall aesthetics of the image. At the same time, the tag layout often lacks adaptability and cannot dynamically adjust according to the blank areas of the image, resulting in a limited reading experience for users.

[0004] 2) Limited interaction methods: Traditional interaction relies heavily on gestures (such as clicking and swiping) or voice commands, which increases the operational burden when users are appreciating the exhibition on their handheld devices, and lacks privacy and convenience in public exhibition scenarios.

[0005] 3) High computational resource consumption: When rendering high-definition interfaces, the entire screen is often rendered indiscriminately under high load, ignoring the foveal characteristics of human vision, which leads to shortened battery life and severe overheating of mobile devices, affecting the immersive experience.

[0006] 4) Insufficient tracking stability: For long scroll paintings or ink paintings with indistinct features, traditional feature point extraction algorithms are easily affected by lighting and texture sparsity, resulting in virtual model jitter or loss.

[0007] Therefore, there is an urgent need for a method that can intelligently avoid occlusion, has adaptive layout capabilities, provides natural interaction, and is computationally efficient in displaying calligraphy and painting in a virtual-real fusion manner. Summary of the Invention

[0008] This invention provides a method and apparatus for displaying calligraphy and paintings using augmented reality (AR). This invention solves the problems of AR tags obscuring the image, rigid layout, cumbersome interaction, and low rendering efficiency in existing technologies.

[0009] In a first aspect, embodiments of the present invention provide a method for displaying calligraphy and painting using augmented reality, the method comprising: In an augmented reality eye-tracking digital terminal, the display of calligraphy and painting appreciation works is scanned, and a two-layer data architecture is constructed, consisting of a dense geometric layer and a sparse tracking layer. The dense geometric layer includes a dense three-dimensional point cloud, and the sparse tracking layer includes a sparse spatial map. Extract information anchor points of specific elements in the display of calligraphy and painting works, and bind them with the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets; Based on dense 3D point cloud and sparse spatial map, eye tracking and gaze point calculation are performed on the user to obtain accurate gaze point coordinates. Based on the information anchor point set, hit determination and activation are performed to obtain the target digital information data packet and target 3D polygon boundary of the target anchor point hit by the user's gaze. Based on dense 3D point clouds, the substrate and potential field environment are constructed according to the precise line-of-sight coordinates and the boundary of the target 3D polygon. Then, a multi-objective enzyme-based optimization algorithm is launched to solve the user's 3D pose coordinates. Based on three-dimensional pose coordinates, the target digital information data packet is dynamically superimposed and rendered on the display screen of calligraphy and painting appreciation works, realizing the virtual and real integration display of calligraphy and painting.

[0010] The technical solution provided in this application has at least the following beneficial effects: By introducing a hybrid repulsive potential field and a multi-target enzyme action optimization algorithm, the system can automatically calculate the optimal label placement position based on the ink distribution (static obstacle) and existing AR labels (dynamic obstacle) in the calligraphy and painting, achieving "unobstructed" display and maximizing the protection of the integrity of the calligraphy and painting. By introducing eye-tracking angular velocity and S-shaped membership functions, the system can identify the user's "deep reading" and "rapid scanning" states and dynamically adjust the label attachment distance, ensuring convenience during deep reading and avoiding visual interference caused by labels during rapid scanning. Based on eye-tracking concave rendering technology, the system performs full-resolution rendering of the gaze area and downsampling rendering of the non-gaze area, significantly reducing GPU load and improving device battery life while ensuring visual clarity. A dual-layer data architecture is adopted, with a dense geometry layer providing accurate geometric information and a sparse tracking layer using highly robust feature points selected by fractal dimension for pose tracking, effectively solving the tracking loss problem caused by the sparse texture of ink paintings.

[0011] In one alternative implementation, an augmented reality eye-tracking digital terminal scans the displayed image of a calligraphy and painting appreciation work and constructs a two-layer data architecture consisting of a dense geometric layer and a sparse tracking layer, including: In an augmented reality eye-tracking digital terminal, the terminal's depth sensor is used to scan the display of calligraphy and painting works to obtain the original depth map of the display. Align the original depth map of the artwork display with the corresponding RGB image to generate a dense three-dimensional point cloud containing spatial coordinates and color information, which serves as a dense geometric layer of a two-layer data architecture. Several candidate key points are extracted from the RGB image. A filtering threshold is introduced to filter the candidate key points, resulting in a number of sparse key points that are retained. A corresponding sparse spatial map is then generated as the sparse tracking layer of the two-layer data architecture.

[0012] In one optional implementation, several candidate keypoints are extracted from the RGB image. A filtering threshold is introduced to filter these candidate keypoints, resulting in a retained set of sparse keypoints. A corresponding sparse spatial map is then generated as the sparse tracking layer of a two-layer data architecture, including: Using an improved ORB algorithm, several candidate keypoints are extracted from an RGB image. The fractal dimension of the local image patch for each candidate keypoint is calculated using the differential box counting method. Candidate key points of local image patches with fractal dimensions greater than the filtering threshold are retained, and the remaining candidate key points are removed to obtain a number of sparse key points. The retained sparse key points are combined with the corresponding depths in the original depth map, and back-projected to generate a sparse spatial map with three-dimensional coordinates and descriptors, which serves as the sparse tracing layer of a two-layer data architecture.

[0013] In one optional implementation, information anchor points of specific elements in the displayed image of a calligraphy and painting appreciation work are extracted and bound to the corresponding digital information data packet, generating a set of information anchor points including the three-dimensional polygon boundary and the digital information data packet, including: On a high-precision image of a work of calligraphy and painting appreciation, specific elements of the work are selected by using a labeling tool, and the two-dimensional polygonal boundary of the specific elements in the image pixel coordinate system is obtained. Based on the original depth map of the dense 3D point cloud of the dense geometry layer, the 2D polygon boundary is back-projected onto the 3D physical space to obtain the 3D polygon boundary, and the centroid of the 3D polygon boundary is used as the information anchor point. The information anchor points are bound to the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets. The digital information data packets include expert interpretation texts of specific elements, seal interpretation texts, simplified and traditional Chinese texts, and UI type flags.

[0014] In one optional implementation, eye tracking and gaze point calculation are performed on the user based on dense 3D point clouds and sparse spatial maps to obtain precise gaze point coordinates. Then, based on a set of information anchor points, hit determination and activation are performed to obtain the target digital information data packet of the target anchor point hit by the user's gaze and the target 3D polygon boundary, including: In the eye-tracking digital terminal, the terminal camera is used to collect environmental images during operation, and the sparse environmental visual features of the current frame of the calligraphy and painting appreciation artwork display screen are extracted from the environmental images. The current frame's sparse environment visual features are matched with the sparse spatial map of the sparse tracking layer, and the PnP problem is solved by minimizing the reprojection error to obtain the six-degree-of-freedom pose at the current moment. Using an eye-tracking device, the user's pupil center and corneal reflection point are collected, the gaze unit vector in the camera coordinate system is calculated, and the gaze unit vector is converted to the gaze ray in the world coordinate system based on the six-degree-of-freedom pose. Based on the dense 3D point cloud of the dense geometry layer, a KD-Tree is constructed, and the line of sight is input into the index of the KD-Tree to find the nearest intersection point and obtain the precise coordinates of the line of sight landing point; Calculate the Euclidean distance from the precise gaze point coordinates to each information anchor point in the information anchor point set. If the judgment condition is met, determine that the gaze hits the information anchor point and take the information anchor point as the target anchor point hit by the user's gaze. Extract the target digital information data packet and the target 3D polygon boundary of the target anchor point hit by the user's line of sight.

[0015] In one alternative implementation, based on a dense 3D point cloud, a substrate and potential field environment is constructed according to the precise coordinates of the line-of-sight point and the boundary of the target 3D polygon. A multi-objective enzyme-based optimization algorithm is then initiated to solve for the user's 3D pose coordinates, including: A candidate solution space for a semi-cylindrical surface is constructed with the precise coordinates of the line of sight as the center. The set of ink marks pixels of the whole picture is extracted from the dense three-dimensional point cloud and combined with the boundary of the target three-dimensional polygon to construct a hybrid repulsive potential field. The user's three-dimensional pose coordinates are encoded into the position vector of the enzyme molecule in the multi-objective enzyme action optimization algorithm, and the fitness function of the multi-objective enzyme action optimization algorithm is constructed based on the mixed repulsion potential field and the precise line-of-sight landing point coordinates. Based on the fitness function, a multi-objective enzyme action optimization algorithm is used to perform iterative optimization of enzyme molecular allosteric transformation to obtain the Pareto optimal solution set. The first derivative of eye movement is extracted to obtain the line-of-sight angular velocity. An S-shaped membership degree is constructed, and extreme solutions are selected from the Pareto optimal solution set and weighted and fused to obtain the user's final three-dimensional pose coordinates.

[0016] In one alternative implementation, the fitness function is formulated as follows: In the formula, enzyme molecules X The fitness values ​​of the corresponding alternative 3D pose coordinates; enzyme molecules X The specific alignment penalty value of the corresponding alternative three-dimensional pose coordinates; For individuals X The cognitive load penalty value of the corresponding alternative three-dimensional pose coordinates; In the formula, enzyme molecules 3D coordinates of the center point label ; To accurately determine the coordinates of the line of sight In the candidate solution space Three-dimensional mapped coordinates on; enzyme molecules X The Euclidean distance between the center point and the precise coordinates of the line of sight; enzyme molecules The plane rotation angle; for A mixed repulsive potential field; For safe levitation depth; These are the multi-dimensional cognitive weighting coefficients.

[0017] In one alternative implementation, a multi-objective enzyme action optimization algorithm is used to iteratively search for enzyme molecular allosteric transformations based on a fitness function, yielding a Pareto optimal solution set, including: The chaotic sequence is generated using Logistic mapping, and then mapped to the parameter space of the enzyme molecule in the multi-objective enzyme action optimization algorithm to obtain the initial population. Using the fitness function, the fitness value of each initial enzyme molecule in the initial population is calculated, and based on the fitness value, the optimal position of the enzyme molecule and the global optimal enzyme molecule are determined. Based on the optimal position of the enzyme molecule and the global optimal enzyme molecule, calculate the initial basic movement step size of the enzyme molecule and the difference between the potential field value at the pre-updated position and the potential field value at the current position. If the difference is greater than the potential field surge threshold, trigger the allosteric change and proceed to the next step. The basic moving step size is decomposed into a penetrating component along the potential field gradient direction and a perpendicular component perpendicular to the gradient direction. Based on the difference in potential field surge, the morphological contraction coefficient and expansion coefficient are dynamically calculated. The initial population is updated based on the penetration component, vertical component, allosteric contraction coefficient, and expansion coefficient to obtain the updated population. Using a fitness function, the fitness value of each updated enzyme molecule in the updated population is calculated. Based on the fitness value, non-dominated sorting and crowding calculation are performed, dominated inferior solutions are eliminated, and the frontier solution set is output as the Pareto optimal solution set.

[0018] In one optional implementation, based on three-dimensional pose coordinates, the target digital information data packet is dynamically overlaid and rendered onto the display screen of the calligraphy and painting appreciation work to achieve a virtual-real fusion display of calligraphy and painting, including: Parse the target digital information data packet to obtain expert interpretation text, seal interpretation text, simplified and traditional Chinese text comparison text, and UI type flag bits for specific elements of the target. Based on the UI type flag bits, call the corresponding preset UI template and construct a three-dimensional UI bounding mesh composed of polygon vertices in local memory. Extract the final 3D coordinates and final planar rotation angle of the center point label in the 3D pose coordinate system, and construct the UI world transformation matrix in the world coordinate system. Obtain the inverse matrix and intrinsic parameter matrix for calculating the six-degree-of-freedom pose, and cascade the local mesh vertices to screen normalized coordinates; The screen normalized coordinates are projected into two-dimensional screen coordinates, and the roll angle component extracted from the inverse matrix of the six-degree-of-freedom pose is used to perform inverse affine compensation on the two-dimensional screen coordinates to obtain the horizontal correction coordinates. By iterating through the screen-normalized coordinates of all geometric vertices and performing inverse affine compensation, we obtain the set of boundary coordinates of all geometric vertices mapped to the screen's two-dimensional space, as well as the discrete vertex depth values ​​corresponding to the geometric vertices under the camera's view frustum, thus forming the geometric skeleton for subsequent pixel-level rendering. Based on the geometric skeleton, primitive assembly and scanning transformation are performed according to the boundary coordinate set to determine the two-dimensional pixel area covered by the UI on the screen. For any screen pixel to be rendered within the two-dimensional pixel area, linear interpolation calculation is performed on the discrete vertex depth value using the proportion of the centroid coordinates of the pixel inside the polygon to obtain the virtual interpolation depth corresponding to the pixel. Based on the original depth map of the three-dimensional pose coordinates and the dense three-dimensional point cloud, a real scene depth map of the current frame is generated. The pixel-level depth difference between the virtual interpolation depth and the real scene depth map is calculated. The pixel-level depth difference is substituted into the transparency decay function to calculate the depth cancellation transparency coefficient corresponding to each pixel in the two-dimensional pixel region, thereby realizing the soft cancellation of depth conflicts in the virtual UI. The gaze unit vector is projected onto the screen of the eye-tracking digital terminal to obtain the two-dimensional gaze point coordinates. For the two-dimensional pixel area, the Euclidean distance from any pixel to the gaze point is calculated. Based on the foveal vision mechanism, the dynamic downsampling factor is calculated. Using the GPU of the eye-tracking digital terminal, the gaze center area of ​​the effective UI pixel area is rendered at 1:1 full resolution according to the dynamic downsampling factor, while the edge leading-out area of ​​the effective UI pixel area is rendered at low resolution. According to the transparency attenuation parameter, the material background color of the expert interpretation text, seal interpretation text and simplified / traditional text in the effective UI pixel area is weighted and fused with the physical background color of the eye-tracking digital terminal to generate a virtual and real fusion image of calligraphy and painting, which is then displayed.

[0019] Secondly, embodiments of the present invention provide an augmented reality-based calligraphy and painting virtual-real fusion display device for realizing a calligraphy and painting virtual-real fusion display method, the device comprising: The spatial perception unit is used to scan the display of calligraphy and painting appreciation works on an augmented reality eye-tracking digital terminal, and to construct a dense geometric layer and a sparse tracking layer of a two-layer data architecture. The dense geometric layer includes a dense three-dimensional point cloud, and the sparse tracking layer includes a sparse spatial map. The information anchor point generation unit is used to extract information anchor points of specific elements in the display of calligraphy and painting appreciation works, and bind them with the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets. The eye-tracking unit is used to perform eye tracking and gaze point calculation on the user based on dense 3D point cloud and sparse spatial map, to obtain accurate gaze point coordinates, and to perform hit determination and activation based on information anchor point set, to obtain target digital information data packet and target 3D polygon boundary of the target anchor point hit by the user's gaze. The optimized rendering unit is used to construct the substrate and potential field environment based on dense 3D point clouds, according to the precise coordinates of the viewpoint and the boundary of the target 3D polygon, and to start a multi-objective enzyme-based optimization algorithm to solve the user's 3D pose coordinates. The virtual-real fusion display unit is used to dynamically overlay and render target digital information data packets onto the display screen of calligraphy and painting appreciation works based on three-dimensional pose coordinates, so as to realize the virtual-real fusion display of calligraphy and painting.

[0020] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of a method for displaying calligraphy and painting using augmented reality based on an embodiment of the present invention. Figure 3 This is a schematic diagram of the functional units of a virtual-real fusion display device for calligraphy and painting based on augmented reality, provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0024] The present invention will be further described below with reference to the accompanying drawings.

[0025] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0026] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0027] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0028] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for an augmented reality-based virtual-real fusion display device for calligraphy and painting.

[0029] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the augmented reality-based calligraphy and painting virtual and real fusion display device stored in the memory 1005 through the processor 1001, and executes the augmented reality-based calligraphy and painting virtual and real fusion display method provided in the embodiment of the present invention.

[0030] Reference Figure 2 The embodiments of the present invention provide a method for displaying calligraphy and painting using augmented reality, the method comprising: S201: In an augmented reality eye-tracking digital terminal, the display screen of calligraphy and painting appreciation works is scanned, and a two-layer data architecture of dense geometry layer and sparse tracking layer is constructed. The dense geometry layer includes dense three-dimensional point cloud, and the sparse tracking layer includes sparse spatial map. S202: Extract information anchor points of specific elements in the display of calligraphy and painting works, and bind them with the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets; S203: Based on dense 3D point cloud and sparse spatial map, eye tracking and gaze point calculation are performed on the user to obtain accurate gaze point coordinates. Based on the information anchor point set, hit determination and activation are performed to obtain the target digital information data packet and target 3D polygon boundary of the target anchor point hit by the user's gaze. S204: Based on dense 3D point cloud, construct the substrate and potential field environment according to the precise line-of-sight coordinates and the target 3D polygon boundary, and start a multi-objective enzyme action optimization algorithm to solve the user's 3D pose coordinates; S205: Based on three-dimensional pose coordinates, the target digital information data packet is dynamically superimposed and rendered on the display screen of calligraphy and painting appreciation works to achieve a virtual-real integrated display of calligraphy and painting.

[0031] The technical solution provided in this application has at least the following beneficial effects: By introducing a hybrid repulsive potential field and a multi-target enzyme action optimization algorithm, the system can automatically calculate the optimal label placement position based on the ink distribution (static obstacle) and existing AR labels (dynamic obstacle) in the calligraphy and painting, achieving "unobstructed" display and maximizing the protection of the integrity of the calligraphy and painting. By introducing eye-tracking angular velocity and S-shaped membership functions, the system can identify the user's "deep reading" and "rapid scanning" states and dynamically adjust the label attachment distance, ensuring convenience during deep reading while avoiding visual interference caused by labels during rapid scanning. Based on eye-tracking concave rendering technology, the system performs full-resolution rendering of the gaze area and downsampling rendering of the non-gaze area, significantly reducing the GPU load while ensuring visual clarity and improving the device's battery life. A dual-layer data architecture is adopted, with a dense geometry layer providing accurate geometric information and a sparse tracking layer using highly robust feature points selected by fractal dimension for pose tracking, effectively solving the tracking loss problem caused by the sparse texture of ink paintings.

[0032] In one alternative implementation, an augmented reality eye-tracking digital terminal scans the displayed image of a calligraphy and painting appreciation work and constructs a two-layer data architecture consisting of a dense geometric layer and a sparse tracking layer, including: S2011: In an augmented reality eye-tracking digital terminal, the terminal's depth sensor is used to scan the display of calligraphy and painting works to obtain the original depth map of the display of calligraphy and painting works. S2012: Align the original depth map of the artwork display with the corresponding RGB image to generate a dense three-dimensional point cloud containing spatial coordinates and color information, which serves as a dense geometric layer of a two-layer data architecture. S2013: Extract several candidate key points from the RGB image, introduce a filtering threshold, filter the candidate key points to obtain several sparse key points, and generate a corresponding sparse spatial map as the sparse tracking layer of the two-layer data architecture.

[0033] In one optional implementation, several candidate keypoints are extracted from the RGB image. A filtering threshold is introduced to filter these candidate keypoints, resulting in a retained set of sparse keypoints. A corresponding sparse spatial map is then generated as the sparse tracking layer of a two-layer data architecture, including: S20131: Using an improved Oriented Fast and Rotated BRIEF (ORB) algorithm, extract several candidate keypoints from an RGB image; In this embodiment, the improved ORB algorithm aims to quickly and stably extract sparse keypoints with scale and rotation invariance from RGB images. It consists of two core parts and has been optimized for calligraphy and painting scenarios. Feature point detection: Improved Oriented Features from Accelerated Segment Test (oFAST) algorithm: This stage is responsible for identifying potential corner or edge points in the image; The standard FAST principle is as follows: 16 pixels are selected on a circle with a radius of 3, centered on pixel P; if the gray values ​​of 12 consecutive pixels are significantly higher or lower than the gray value of point P, then P is considered a feature point; this is a very fast corner detector. Challenges and improvements in calligraphy and painting scenarios: The standard FAST detector detects a large number of repetitive, low-contrast points on the artwork, leading to subsequent matching problems; improvements include: Adaptive FAST thresholding: Fixed thresholds are no longer used; the detection threshold is dynamically adjusted based on the pixel grayscale statistics of local image regions; high thresholds are used to refine points in areas with dense ink and strong contrast (such as mountain and rock outlines), while low thresholds are used to ensure sufficient key points in areas with uniform rendering and low contrast (such as large areas of sky and water). A secondary screening using fractal dimension is introduced: among the candidate points detected by FAST, the surface fractal dimension of each point and its neighborhood is calculated, and only points with FD higher than the threshold are retained. The fractal dimension can effectively characterize the complexity and irregularity of the ink edge, thereby filtering out invalid feature points generated by smooth and homogeneous background areas, greatly improving the uniqueness and stability of feature points. Uniform distribution control (quadtree / grid method): To ensure that feature points are evenly distributed across the entire image (which is crucial for large-format SLAM tracking), a quadtree algorithm or adaptive grid method is used to filter and redistribute the detected feature points; the image region is recursively segmented, and the feature point with the highest response value is retained in each sub-region to avoid feature point clustering. Feature descriptor: Rotation-Aware BRIEF (rBRIEF) algorithm: This stage generates a "digital fingerprint" describing the local image pattern of the detected key points; The standard BRIEF principle: Select N pairs (e.g., 256 pairs) of pixels within the image patch around the key point; compare the grayscale values ​​of each pair of points to generate an N-bit binary string, which is extremely fast. The drawback of standard BRIEF is that it is very sensitive to image rotation. ORB Improvements - rBRIEF: Calculate the principal orientation of feature points: Assign an orientation angle to each key point using the gray-scale centroid method, which gives the descriptor rotation invariance; Generate rotation-aware descriptors: Rotate the N pixel pairs used for comparison according to the principal direction so that the descriptors remain stable even if the image is rotated; S20132: Using the differential box counting method, calculate the fractal dimension of the local image patch for each candidate keypoint, using the following formula: In the formula, For the first j The fractal dimension of a local image patch containing candidate keypoints; j Indexing candidate key points; For measurement scale; To cover the number of boxes, i.e., at a specific measurement scale The minimum number of boxes required to completely cover all valid pixels (i.e., ink blot pixels with grayscale values ​​higher than the background threshold) within a local image patch, as... Increase the number of boxes required It will inevitably decrease; S20133: Retain candidate key points of local image blocks whose fractal dimension is greater than the filtering threshold, and remove the remaining candidate key points to obtain a number of sparse key points; S20134: Combine the retained sparse key points with the corresponding depths in the original depth map, and backproject to generate a sparse spatial map with three-dimensional coordinates and descriptors, which serves as the sparse tracking layer of a two-layer data architecture.

[0034] In one optional implementation, information anchor points of specific elements in the displayed image of a calligraphy and painting appreciation work are extracted and bound to the corresponding digital information data packet, generating a set of information anchor points including the three-dimensional polygon boundary and the digital information data packet, including: S2021: On a high-precision image of a work of calligraphy and painting appreciation, a specific element of the work of calligraphy and painting appreciation is selected by using a labeling tool, and the two-dimensional polygonal boundary of the specific element in the image pixel coordinate system is obtained. S2022: Based on the original depth map of the dense three-dimensional point cloud of the dense geometry layer, the two-dimensional polygon boundary is back-projected to the three-dimensional physical space to obtain the three-dimensional polygon boundary, and the centroid of the three-dimensional polygon boundary is used as the information anchor point. S2023: Bind the information anchor points to the corresponding digital information data packets to generate an information anchor point set including three-dimensional polygon boundaries and digital information data packets. The digital information data packets include expert interpretation texts of specific elements, seal interpretation texts, simplified and traditional Chinese texts, and UI type flags.

[0035] In one optional implementation, eye tracking and gaze point calculation are performed on the user based on dense 3D point clouds and sparse spatial maps to obtain precise gaze point coordinates. Then, based on a set of information anchor points, hit determination and activation are performed to obtain the target digital information data packet of the target anchor point hit by the user's gaze and the target 3D polygon boundary, including: S2031: In the eye-tracking digital terminal, the terminal camera is used to collect environmental images during operation, and the sparse environmental visual features of the current frame of the calligraphy and painting appreciation artwork display screen in the environmental images are extracted. S2032: Perform feature matching between the sparse environment visual features of the current frame and the sparse spatial map of the sparse tracking layer, and solve the perspective-n-point (PnP) problem by minimizing the reprojection error to obtain the six-degree-of-freedom pose at the current moment, as shown in the formula: In the formula, For a moment l The six degrees of freedom pose refers to the pose of the terminal camera at any time. lThe strict spatial transformation matrix relative to the world coordinate system (i.e., the initial coordinate system when building the map); For a moment l The rotation matrix, a 3×3 orthogonal matrix, represents the current "orientation" of the terminal camera in three-dimensional space (including yaw angle, pitch angle, and roll angle). As a translation vector, a 3×13×1 column vector, representing the "absolute coordinate position" of the optical center of the terminal camera in three-dimensional space; l The current moment; S2033: Using an eye-tracking device, the user's pupil center and corneal reflection point are collected. The gaze unit vector in the camera coordinate system is calculated, and based on the six-degree-of-freedom pose, the gaze unit vector is transformed into a gaze ray in the world coordinate system. The formula is as follows: In the formula, Let be the line of sight in the world coordinate system, and let be the parametric equation of an infinitely long semi-straight line in the three-dimensional physical world. Let be a parameter variable (scalar) of the ray, representing the physical distance from the optical center along the line of sight, and the constraints. >0 indicates that the ray only extends in front of the camera and does not extend backward; The line-of-sight unit vector is calculated by capturing the relative geometric relationship between the pupil center and the corneal reflection point using the built-in eye-tracking camera. It is a 3×13×1 column vector with a magnitude of 1. At this time, the reference origin of the vector is the optical center of the camera, and the reference axis of the direction is the image coordinate system of the camera (X to the right, Y down, Z forward). S2034: Based on the dense three-dimensional point cloud of the dense geometric layer, construct a K-Dimensional Tree (KD-Tree), input the line of sight ray into the index of the KD-Tree, find the nearest intersection point, and obtain the precise coordinates of the line of sight landing point; S2035: Calculate the Euclidean distance from the precise gaze point coordinates to each information anchor point in the information anchor point set. If the judgment condition is met, determine that the gaze has hit the information anchor point, and take the information anchor point as the target anchor point hit by the user's gaze. The formula for the judgment condition is: In the formula, To accurately determine the coordinates of the line of sight; For the first k The centroid coordinates of the information anchor point; k For information anchor indexing; The Euclidean distance threshold; The function is for finding the minimum value; S2036: Extract the target digital information data packet and the target three-dimensional polygon boundary of the target anchor point hit by the user's line of sight.

[0036] In one alternative implementation, based on a dense 3D point cloud, a substrate and potential field environment is constructed according to the precise coordinates of the line-of-sight point and the boundary of the target 3D polygon. A multi-objective enzyme-based optimization algorithm is then initiated to solve for the user's 3D pose coordinates, including: S2041: Construct a candidate solution space for a semi-cylindrical surface centered on the precise coordinates of the line-of-sight point. Extract the entire image's ink pixel set from the dense 3D point cloud and combine it with the target 3D polygon boundary to construct a hybrid repulsive potential field. The formula is: In the formula, For the mixed repulsive potential function, for the candidate solution space any point on The larger the value calculated by substituting it into the formula, the more "dangerous" the position (closer to ink or other labels), and the more likely the algorithm will avoid that position. The smaller the value (closer to 0), the more "safe" the position (in blank space and unobstructed), representing the optimal solution sought by the algorithm. The coordinates of the candidate solution are denoted as , where is the coordinates of any discrete grid point on the semi-cylindrical surface, representing the position of a potential AR tag rendering anchor point. The candidate solution space for the semi-cylindrical surface is a three-dimensional semi-cylindrical surface area of ​​a preset size that protrudes outward from the outside of the painting / calligraphy surface (i.e. towards the user) with the precise coordinates of the viewpoint as the center. This simulates the legal floating range of the AR tag in real space (it cannot penetrate walls, and it cannot be too close or too far from the painting). The set of ink pixels in the whole picture is a three-dimensional point set of pure ink area extracted by filtering the points in the dense three-dimensional point cloud by color / grayscale threshold. It constitutes a "static insurmountable mountain" in the potential field. Any point that falls into this area will be subjected to a strong repulsive force to prevent the label from obscuring the calligraphy and painting. The target is a three-dimensional polygon boundary, and the set of three-dimensional bounding boxes (or polygon boundaries) of other AR tags that have been activated and displayed on the screen constitutes a "dynamic moving roadblock" in the potential field to prevent new tags from spatially overlapping with old tags. for The coordinates of a specific 3D point in the ink blot; for The three-dimensional boundary geometry of a specific existing label in the text; A static Gaussian repulsion kernel for ink stains, constructed for the exponential function, simulates the "flexible magnetic repulsion" generated by calligraphy and painting ink stains on the label. The repulsion force at the edge of the ink stain is not abrupt 0 and 1, but a smooth transition, which ensures that the gradient is continuous during the optimization process and will not get stuck in deadlock. The hyperparameter (in centimeters) is the Gaussian kernel bandwidth, which controls the decay rate of the ink repulsion force. The value of determines the width of the "safety buffer zone" between the label and the edge of the ink. The larger the ink, the slower the repulsive force decays, and the farther the label is pushed away from the ink. The smaller the size, the closer the label can be placed; This is a dynamic label rejection function that handles anti-overlap logic between multiple AR labels. When a candidate point falls inside the boundary or boundary buffer of an existing label, this function outputs a high penalty value. It is typically defined as a distance-based step function or a linear piecewise function, for example: when... lie in When it is outside and the distance is greater than the safety clearance, =0; when invade When within the safe distance, As the depth of intrusion increases linearly or exponentially, the repulsion between labels is typically rigid (not allowing any volumetric overlap), unlike the Gaussian repulsion of the first term. S2042: Encode the user's three-dimensional pose coordinates into the position vector of the enzyme molecule in the multi-objective enzyme action optimization algorithm, and construct the fitness function of the multi-objective enzyme action optimization algorithm based on the mixed repulsion potential field and the precise line-of-sight landing point coordinates. S2043: Based on the fitness function, a multi-objective enzyme action optimization algorithm is used to perform iterative optimization of enzyme molecular allosteric transformation to obtain the Pareto optimal solution set; S2044: Extract the first derivative of eye movement to obtain the line-of-sight angular velocity, construct an S-shaped membership system, and combine the S-shaped membership system with extreme solutions selected from the Pareto optimal solution set for weighted fusion to obtain the user's final 3D pose coordinates. The formula is as follows: In the formula, These are the final three-dimensional pose coordinates; The final three-dimensional coordinates of the center point label; This is the final planar rotation angle; The extremely far avoidance solution selected from the Pareto optimal solution set is the solution that is "farthest from the point of view (but within a safe range) and avoids the most thoroughly". Its characteristics are absolute safety and no obstruction of ink, but it requires the user's line of sight to be significantly shifted. The closest solution selected from the Pareto optimal solution set is the solution that is "closest to the point of view". Its characteristics are that the reading line of sight is shifted very little, but it may be slightly closer to the edge of the screen or there may be a slight risk of occlusion. For S-shaped membership, a smooth weight coefficient with a value strictly limited to the range of (0,1) acts as a "fuzzy logic switch", smoothly mapping the continuously changing physical angular velocity to the "trust ratio" of two different UI layout strategies. The slope of the S-curve is a positive control parameter that determines the sensitivity of state switching. The larger the value, the steeper the curve, meaning that the system will immediately and completely switch the display strategy when the user slightly accelerates their scanning. The smaller the value, the flatter the curve, and the smoother the strategy switching. The threshold for cognitive mode switching is the angular velocity value corresponding to the center symmetry point of the S-shaped curve, which is a key empirical threshold based on human factors engineering calibration. The line-of-sight angular velocity is the rate of change of the angle of the user's line of sight in space within an extremely short time window (the unit is usually radians per second, rad / s). It is an exponential function; Scenario A (In-depth reading) →0): The user is staring at a certain stamp (low angular velocity), the formula degenerates into... ≈ The interpretation labels are directly attached next to the stamp, making it easy for users to read related information without moving their eyes, thus reducing the cognitive load of micro-reading; Scene B (Quick scan) →1): The user's gaze rapidly scans left and right across the entire scroll to find areas of interest (high angular velocity), the formula degenerates into... ≈ If the label is placed next to you at this time, it will block your vision and cause dizziness, just like it is pasted on your face. The system will automatically push the label away to the "safety stand" at the edge of your line of sight to ensure the transparency of your vision during the scanning process.

[0037] In one alternative implementation, the fitness function is formulated as follows: In the formula, enzyme molecules X The fitness values ​​of the corresponding alternative 3D pose coordinates; enzyme molecules X The specific alignment penalty value of the corresponding alternative 3D pose coordinates; the smaller the value, the more aligned the label is with the line of sight. For individuals X The smaller the cognitive load penalty value of the corresponding alternative three-dimensional pose coordinates, the less eye strain, less obstruction, and more comfortable the label placement position is; In the formula, enzyme molecules 3D coordinates of the center point label ; To accurately determine the coordinates of the line of sight In the candidate solution space Three-dimensional mapped coordinates on; enzyme molecules X The greater the Euclidean distance between the center point and the precise coordinates of the line of sight, the greater the angle that the user's line of sight needs to shift, resulting in a worse reading experience. enzyme molecules The plane rotation angle refers to the rotation angle of the label of a specific element (such as a text box or speech bubble) when it is placed. In some special layouts of calligraphy and painting, the label of a specific element may need to be slightly tilted to follow the stroke of the brush, rather than always being horizontal. The rotation angle is the ideal plane angle; This is the position-angle weighting coupling coefficient, a hyperparameter used to balance the importance of "distance deviation" and "angle deviation". Since distance deviation usually has a more direct impact on the user experience than angle deviation, it is generally set to a lower value. <1, to suppress excessive amplification of angular deviation in the total penalty; for The mixed repulsive potential field causes the potential field value to surge if the label is placed on the ink, resulting in a huge penalty that forces the algorithm to avoid the core content of the image. This represents the line-of-sight deflection angle, indicating the user's current line-of-sight direction (coordinates from the optical center to the precise point of view). (The connection) and the user's view of the tag center (light center to) The angle between the lines connecting them; To ensure safe levitation depth, the label should be positioned at the normal distance from the physical surface of the artwork. For multi-dimensional cognitive weight coefficients, satisfying A positive number = 1 is used to adjust the priority of different cognitive dimensions (usually occlusion penalty). maximum); In one alternative implementation, a multi-objective enzyme action optimization algorithm is used to iteratively search for enzyme molecular allosteric transformations based on a fitness function, yielding a Pareto optimal solution set, including: S20431: Use Logistic mapping to generate chaotic sequences, and map these chaotic sequences to the parameter space of enzyme molecules in a multi-objective enzyme action optimization algorithm to obtain the initial population. The formula is as follows: In the formula, For the first n+ 1. nThere are several chaotic variables whose values ​​range from [0, 1]. The stability coefficient is typically 4. This sequence is ergodic and random, ensuring that the initial population is uniformly distributed in the solution space, avoiding getting trapped in local optima, which is superior to traditional random initialization. n Indicator of chaotic variables; In the formula, For the initial population, the first i An initial individual; For the first i One chaotic variable; These are the upper and lower limits of the parameter space; i For individual indicators; S20432: Using the fitness function, calculate the fitness value of each initial enzyme molecule in the initial population, and determine the optimal position of the enzyme molecule and the global optimal enzyme molecule based on the fitness value. S20433: Based on the optimal position of the enzyme molecule and the global optimal enzyme molecule, calculate the initial basic movement step size of the enzyme molecule and the difference between the potential field value at the pre-updated position and the potential field value at the current position. If the difference is greater than the potential field surge threshold, trigger the allosteric change and proceed to the next step. In this embodiment, the initial basic movement step size of the enzyme molecule is calculated based on the optimal position of the enzyme molecule and the globally optimal enzyme molecule, using the following formula: In the formula, For the first t The base movement step size for the next iteration; For the first t The iteration of the ... i A new enzyme molecule, in the first iteration, For the first i An initial enzyme molecule; For the first t The iteration of the ... i The optimal position of an individual enzyme molecule and the globally optimal enzyme molecule; t This represents the current iteration number; The self-cognitive acceleration factor is a positive constant that regulates the step size by which an enzyme molecule moves toward its optimal position in its historical sequence. The larger the enzyme molecule is, the more likely it is to linger near a local optimum (i.e., a previously found good position); The social cognitive acceleration coefficient is a positive constant that regulates the step size by which enzyme molecules approach the globally optimal molecule. The larger the value, the faster the algorithm converges, but too large a value can lead to premature convergence (i.e., all labels are crowded into the same seemingly best but actually locally optimal position). As a random perturbation vector, it is a random number (or random vector) dynamically generated in the interval [0,1], breaking the absolutely fixed trajectory and giving the enzyme molecule the characteristics of "Brownian motion" in the optimization process, so that it has the ability to explore randomly out of local optimal solutions (such as out of the dead corner in the ink gap). Calculate the initial pre-update position of the enzyme molecule, and calculate the negative gradient direction of the pre-update position under the mixed repulsive potential field (i.e., the opposite direction of the fastest decrease in potential field, pointing towards the outer edge of the ink blot), to obtain the potential field gradient, as shown in the formula: In the formula, For the first t The iteration of the ... i The pre-update position of each updated enzyme molecule; The gradient of the potential field; The gradient direction; The difference between the potential field value at the pre-update position and the potential field value at the current position is calculated using the following formula: In the formula, Potential field value at the pre-update location Potential field value at current position The difference; This represents the potential field value at the pre-update location; This represents the potential field value at the current position. If the difference is greater than the potential surge threshold, it is determined that the enzyme molecule is about to "collide" with the high-density ink region, triggering an allosteric change and entering the step size decomposition step; otherwise, it directly... , For the first t +1 iterations of the first iteration i A newer enzyme molecule; S20434: The basic moving step size is decomposed into a penetration component along the potential field gradient direction and a perpendicular component perpendicular to the gradient direction. Based on the difference in potential field surge, the deformation contraction coefficient and expansion coefficient are dynamically calculated. The formula is as follows: In the formula, This represents the penetration component along the potential field gradient direction; In the formula, The vertical component is perpendicular to the gradient direction; In the formula, These are the coefficients of thermal expansion and contraction. To minimize the value and prevent complete stagnation; This is the shrinkage sensitivity coefficient; The expansion elasticity coefficient is usually... , ∈[0.5,2.0]; The pre-set small positive potential energy threshold is used to filter out minor potential energy fluctuations and prevent the deformation mechanism from being mistakenly triggered due to numerical calculation errors within a safe region. Deformation only occurs when the potential energy surges significantly. > Only then was it determined that a substantial collision had occurred; S20435: Based on the penetration component, vertical component, metamorphic contraction coefficient, and expansion coefficient, the initial population is updated to obtain the updated population, using the following formula: In the formula, For the first t +1 iterations of the first iteration i A newer enzyme molecule; This is the final step size for the structural transformation; S20436: Using the fitness function, calculate the fitness value of each updated enzyme molecule in the updated population, and perform non-dominated sorting and crowding calculation based on the fitness value. Eliminate dominated inferior solutions and output the frontier solution set as the Pareto optimal solution set.

[0038] In one optional implementation, based on three-dimensional pose coordinates, the target digital information data packet is dynamically overlaid and rendered onto the display screen of the calligraphy and painting appreciation work to achieve a virtual-real fusion display of calligraphy and painting, including: S2051: Parse the target digital information data packet to obtain the expert interpretation text, seal interpretation text, simplified and traditional Chinese text comparison text and UI type flag bit of the target specific element, and call the corresponding preset UI template according to the UI type flag bit to construct a UI three-dimensional bounding mesh composed of polygon vertices in local memory. S2052: Extract the center point label from the 3D pose coordinates to obtain the final 3D coordinates and the final planar rotation angle. Construct the UI world transformation matrix in the world coordinate system; S2053: Obtain the inverse matrix and intrinsic parameter matrix for calculating the six-DOF pose, and perform a concatenated transformation of the local mesh vertices to screen-normalized coordinates. The formula is as follows: In the formula, The screen coordinates are normalized coordinates, and the final two-dimensional screen coordinates are obtained by projection. This directly corresponds to the specific pixel position on the screen; For geometric vertex index; The proportional sign is used because the formula calculates homogeneous coordinates on the right side, which need to be divided by the depth factor (i.e., perspective division) before being displayed in screen space. The camera intrinsic parameter matrix is ​​a 3×3 matrix, which is the physical attribute matrix of the camera. It contains the camera's focal length and optical center and is responsible for compressing three-dimensional points in the camera coordinate system into a two-dimensional image through the principle of perspective projection (objects appear larger when closer and smaller when farther away). Six-degree-of-freedom pose The inverse matrix; This is the UI world transformation matrix, a 4×4 homogeneous matrix, which "transfers" the UI from its local coordinate system to a specified position in the real 3D world. Its translation components are derived from the 3D pose coordinates. The final 3D coordinates of the center point label in the image The rotational component is determined by the final planar rotation angle. Decide; These are the homogeneous coordinates of local vertices in the UI. Since the UI template is drawn on a plane by default, its local depth is always 0. S2054: Project normalized screen coordinates into two-dimensional screen coordinates And based on the roll angle component extracted from the inverse matrix of the six-degree-of-freedom pose, the two-dimensional screen coordinates are... Perform inverse affine compensation to obtain the horizontal correction coordinates. The formula is: In the formula, For horizontal correction coordinates; This is the roll angle component; The coordinates of the center point of the screen viewport; By iterating through the screen-normalized coordinates of all geometric vertices and performing inverse affine compensation, we obtain the set of boundary coordinates of all geometric vertices mapped to the screen's two-dimensional space, as well as the discrete vertex depth values ​​corresponding to the geometric vertices under the camera's view frustum, thus forming the geometric skeleton for subsequent pixel-level rendering. S2055: Based on the geometric skeleton, primitive assembly and scanning transformation are performed according to the boundary coordinate set to determine the two-dimensional pixel area covered by the UI on the screen. For any screen pixel to be rendered within the two-dimensional pixel area, linear interpolation calculation is performed on the discrete vertex depth value using the proportion of the centroid coordinates of the pixel inside the polygon to obtain the virtual interpolation depth corresponding to the pixel. S2056: Based on the 3D pose coordinates and the original depth map of the dense 3D point cloud, generate the real scene depth map of the current frame, calculate the pixel-level depth difference between the virtual interpolated depth and the real scene depth map, and substitute the pixel-level depth difference into the transparency decay function to calculate the depth culling transparency coefficient corresponding to each pixel in the 2D pixel region, thereby realizing the soft culling of depth conflicts in the virtual UI. The formula is: In the formula, For pixels The depth of the hidden transparency coefficient; The threshold for fatality due to molding; The safe levitation threshold; The depth difference is at the pixel level; For pixels The virtual interpolation depth; For pixels in the depth map of the real scene Real-world scene depth; The alpha value is a Gaussian decay coefficient. When the virtual label is too close, the alpha value smoothly drops to 0, creating a three-dimensional visual depth effect where the label "floats" out of the picture without obscuring any real ink marks. S2057: Transform the line-of-sight unit vector Projecting the coordinates of the gaze point onto the screen of the eye-tracking digital terminal, we obtain the two-dimensional gaze point coordinates. For the two-dimensional pixel region, we calculate the Euclidean distance from any pixel to the gaze point. Based on the foveal vision mechanism, we calculate the dynamic downsampling factor using the following formula: In the formula, For pixels The dynamic downsampling factor; This represents the maximum downsampling factor. The attenuation coefficient; The Euclidean distance from the pixel to the gaze point; S2058: Using the GPU of the eye-tracking digital terminal, based on a dynamic downsampling factor, the gaze center area of ​​the effective UI pixel region is rendered at 1:1 full resolution, while the edge leading-out areas of the effective UI pixel region are rendered at low resolution. Based on the transparency attenuation parameter, the material background color of the expert interpretation text, seal inscriptions, and simplified / traditional text in the effective UI pixel region of the calligraphy and painting appreciation artwork is weighted and fused with the physical background color of the eye-tracking digital terminal to generate a calligraphy and painting fusion image, which is then displayed. The formula is: In the formula, To integrate the real and the virtual in the painting and calligraphy, and the pixels in the picture The final output color; The weighted fusion coefficient; For pixels The material's base color; For pixels The physical background color; Transparency is used for material transparency and depth testing.

[0039] This invention also provides an augmented reality-based calligraphy and painting virtual-real fusion display device 300, referring to... Figure 3 The device may include the following units: The spatial perception unit 301 is used to scan the display screen of calligraphy and painting appreciation works on an augmented reality eye-tracking digital terminal, and construct a dense geometric layer and a sparse tracking layer of a two-layer data architecture. The dense geometric layer includes a dense three-dimensional point cloud, and the sparse tracking layer includes a sparse spatial map. The information anchor point generation unit 302 is used to extract information anchor points of specific elements in the display of calligraphy and painting appreciation works, and bind them with the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets. The eye-tracking unit 303 is used to perform eye tracking and gaze point calculation on the user based on dense 3D point cloud and sparse spatial map, to obtain accurate gaze point coordinates, and to perform hit determination and activation based on information anchor point set, to obtain target digital information data packet and target 3D polygon boundary of the target anchor point hit by the user's gaze. The rendering unit 304 is optimized to construct the substrate and potential field environment based on dense 3D point clouds, according to the precise line-of-sight coordinates and the boundary of the target 3D polygon, and to start a multi-objective enzyme-based optimization algorithm to solve the user's 3D pose coordinates. The virtual-real fusion display unit 305 is used to dynamically overlay and render the target digital information data packet onto the display screen of calligraphy and painting appreciation works based on three-dimensional pose coordinates, so as to realize the virtual-real fusion display of calligraphy and painting.

[0040] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; When the processor executes the program stored in the memory, it implements the augmented reality-based method for displaying calligraphy and painting in a virtual-real fusion manner.

[0041] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EI) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0042] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0043] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the augmented reality-based virtual-real fusion display method for calligraphy and painting according to embodiments of the present invention.

[0044] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0045] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0047] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0049] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for displaying calligraphy and painting using augmented reality, characterized in that, The method includes: In an augmented reality eye-tracking digital terminal, the display of calligraphy and painting appreciation works is scanned, and a two-layer data architecture is constructed, consisting of a dense geometric layer and a sparse tracking layer. The dense geometric layer includes a dense three-dimensional point cloud, and the sparse tracking layer includes a sparse spatial map. Extract information anchor points of specific elements in the display of calligraphy and painting works, and bind them with the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets; Based on dense 3D point cloud and sparse spatial map, eye tracking and gaze point calculation are performed on the user to obtain accurate gaze point coordinates. Based on the information anchor point set, hit determination and activation are performed to obtain the target digital information data packet and target 3D polygon boundary of the target anchor point hit by the user's gaze. Based on dense 3D point clouds, the substrate and potential field environment are constructed according to the precise line-of-sight coordinates and the boundary of the target 3D polygon. Then, a multi-objective enzyme-based optimization algorithm is launched to solve the user's 3D pose coordinates. Based on three-dimensional pose coordinates, the target digital information data packet is dynamically superimposed and rendered on the display screen of calligraphy and painting appreciation works, realizing the virtual and real integration display of calligraphy and painting.

2. The method for displaying calligraphy and painting using augmented reality based on claim 1, characterized in that, In an augmented reality eye-tracking digital terminal, the displayed images of calligraphy and painting works are scanned, and a two-layer data architecture consisting of a dense geometric layer and a sparse tracking layer is constructed, including: In an augmented reality eye-tracking digital terminal, the terminal's depth sensor is used to scan the display of calligraphy and painting works to obtain the original depth map of the display. Align the original depth map of the artwork display with the corresponding RGB image to generate a dense three-dimensional point cloud containing spatial coordinates and color information, which serves as the dense geometric layer of the two-layer data architecture. Several candidate key points are extracted from the RGB image. A filtering threshold is introduced to filter the candidate key points, resulting in a number of sparse key points that are retained. A corresponding sparse spatial map is then generated as the sparse tracking layer of the two-layer data architecture.

3. The method for displaying calligraphy and painting using augmented reality based on claim 2, characterized in that, Several candidate keypoints are extracted from an RGB image. A filtering threshold is introduced to filter these candidate keypoints, resulting in a number of retained sparse keypoints. A corresponding sparse spatial map is then generated, serving as the sparse tracking layer of a two-layer data architecture, including: Using an improved ORB algorithm, several candidate keypoints are extracted from an RGB image. The fractal dimension of the local image patch for each candidate keypoint is calculated using the differential box counting method. Candidate key points of local image patches with fractal dimensions greater than the filtering threshold are retained, and the remaining candidate key points are removed to obtain a number of sparse key points. The retained sparse keypoints are combined with the corresponding depths in the original depth map, and back-projected to generate a sparse spatial map with three-dimensional coordinates and descriptors, which serves as the sparse tracing layer of a two-layer data architecture.

4. The augmented reality-based method for displaying calligraphy and painting with a blend of virtual and real elements as described in claim 3, characterized in that... Extract information anchor points from specific elements displayed in calligraphy and painting appreciation works, and bind them to corresponding digital information data packets to generate a set of information anchor points including 3D polygon boundaries and digital information data packets, including: On a high-precision image of a work of calligraphy and painting appreciation, specific elements of the work are selected by using a labeling tool, and the two-dimensional polygonal boundary of the specific elements in the image pixel coordinate system is obtained. Based on the original depth map of the dense 3D point cloud of the dense geometry layer, the 2D polygon boundary is back-projected onto the 3D physical space to obtain the 3D polygon boundary, and the centroid of the 3D polygon boundary is used as the information anchor point. The information anchor points are bound to the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets. The digital information data packets include expert interpretation texts of specific elements, seal interpretation texts, simplified and traditional Chinese texts, and UI type flags.

5. The augmented reality-based method for displaying calligraphy and painting with a blend of virtual and real elements as described in claim 4, characterized in that, Based on dense 3D point clouds and sparse spatial maps, eye tracking and gaze point calculation are performed on the user to obtain precise gaze point coordinates. Then, based on a set of information anchor points, hit determination and activation are performed to obtain the target digital information data packet and the target 3D polygon boundary of the target anchor point hit by the user's gaze, including: In the eye-tracking digital terminal, the terminal camera is used to collect environmental images during operation, and the sparse environmental visual features of the current frame of the calligraphy and painting appreciation artwork display screen are extracted from the environmental images. The current frame's sparse environment visual features are matched with the sparse spatial map of the sparse tracking layer, and the PnP problem is solved by minimizing the reprojection error to obtain the six-degree-of-freedom pose at the current moment. Using an eye-tracking device, the user's pupil center and corneal reflection point are collected, the gaze unit vector in the camera coordinate system is calculated, and the gaze unit vector is converted to the gaze ray in the world coordinate system based on the six-degree-of-freedom pose. Based on the dense 3D point cloud of the dense geometry layer, a KD-Tree is constructed, and the line of sight is input into the index of the KD-Tree to find the nearest intersection point and obtain the precise coordinates of the line of sight landing point; Calculate the Euclidean distance from the precise gaze point coordinates to each information anchor point in the information anchor point set. If the judgment condition is met, determine that the gaze hits the information anchor point and take the information anchor point as the target anchor point hit by the user's gaze. Extract the target digital information data packet and the target 3D polygon boundary of the target anchor point hit by the user's line of sight.

6. The augmented reality-based method for displaying calligraphy and painting with virtual and real elements according to claim 5, characterized in that, Based on dense 3D point clouds, and according to the precise coordinates of the line-of-sight point and the boundary of the target 3D polygon, a substrate and potential field environment are constructed. A multi-objective enzyme-based optimization algorithm is then initiated to solve for the user's 3D pose coordinates, including: A candidate solution space for a semi-cylindrical surface is constructed with the precise coordinates of the line of sight as the center. The set of ink marks pixels of the whole picture is extracted from the dense three-dimensional point cloud and combined with the boundary of the target three-dimensional polygon to construct a hybrid repulsive potential field. The user's three-dimensional pose coordinates are encoded into the position vector of the enzyme molecule in the multi-objective enzyme action optimization algorithm, and the fitness function of the multi-objective enzyme action optimization algorithm is constructed based on the mixed repulsion potential field and the precise line-of-sight landing point coordinates. Based on the fitness function, a multi-objective enzyme action optimization algorithm is used to perform iterative optimization of enzyme molecular allosteric transformation to obtain the Pareto optimal solution set. The first derivative of eye movement is extracted to obtain the line-of-sight angular velocity. An S-shaped membership degree is constructed, and extreme solutions are selected from the Pareto optimal solution set and weighted and fused to obtain the user's final three-dimensional pose coordinates.

7. The method for displaying calligraphy and painting using augmented reality based on claim 6, characterized in that, The formula for the fitness function is: In the formula, enzyme molecules X The fitness values ​​of the corresponding alternative 3D pose coordinates; enzyme molecules X The specific alignment penalty value of the corresponding alternative three-dimensional pose coordinates; For individuals X The cognitive load penalty value of the corresponding alternative three-dimensional pose coordinates; In the formula, enzyme molecules 3D coordinates of the center point label ; To accurately determine the coordinates of the line of sight In the candidate solution space Three-dimensional mapped coordinates on; enzyme molecules X The Euclidean distance between the center point and the precise coordinates of the line of sight; enzyme molecules The plane rotation angle; for A mixed repulsive potential field; For safe levitation depth; These are the multi-dimensional cognitive weighting coefficients.

8. The method for displaying calligraphy and painting using augmented reality based on claim 7, characterized in that, Based on the fitness function, a multi-objective enzyme action optimization algorithm is used to perform iterative optimization of enzyme molecular allosteric transformation to obtain the Pareto optimal solution set, including: The chaotic sequence is generated using Logistic mapping and then mapped to the parameter space of the enzyme molecule in the multi-objective enzyme action optimization algorithm to obtain the initial population. Using the fitness function, the fitness value of each initial enzyme molecule in the initial population is calculated, and based on the fitness value, the optimal position of the enzyme molecule and the global optimal enzyme molecule are determined. Based on the optimal position of the enzyme molecule and the global optimal enzyme molecule, calculate the initial basic movement step size of the enzyme molecule and the difference between the potential field value at the pre-updated position and the potential field value at the current position. If the difference is greater than the potential field surge threshold, trigger the allosteric change and proceed to the next step. The basic moving step size is decomposed into a penetrating component along the potential field gradient direction and a perpendicular component perpendicular to the gradient direction. Based on the difference in potential field surge, the morphological contraction coefficient and expansion coefficient are dynamically calculated. The initial population is updated based on the penetration component, vertical component, allosteric contraction coefficient, and expansion coefficient to obtain the updated population. Using a fitness function, the fitness value of each updated enzyme molecule in the updated population is calculated. Based on the fitness value, non-dominated sorting and crowding calculation are performed, dominated inferior solutions are eliminated, and the frontier solution set is output as the Pareto optimal solution set.

9. The method for displaying calligraphy and painting using augmented reality based on claim 8, characterized in that, Based on three-dimensional pose coordinates, the target digital information data packet is dynamically overlaid and rendered onto the display screen of calligraphy and painting appreciation works, realizing a virtual-real integrated display of calligraphy and painting, including: Parse the target digital information data packet to obtain expert interpretation text, seal interpretation text, simplified and traditional Chinese text comparison text, and UI type flag bits for specific elements of the target. Based on the UI type flag bits, call the corresponding preset UI template and construct a three-dimensional UI bounding mesh composed of polygon vertices in local memory. Extract the final 3D coordinates and final planar rotation angle of the center point label in the 3D pose coordinate system, and construct the UI world transformation matrix in the world coordinate system. Obtain the inverse matrix and intrinsic parameter matrix for calculating the six-degree-of-freedom pose, and cascade the local mesh vertices to screen normalized coordinates; The screen normalized coordinates are projected into two-dimensional screen coordinates, and the roll angle component extracted from the inverse matrix of the six-degree-of-freedom pose is used to perform inverse affine compensation on the two-dimensional screen coordinates to obtain the horizontal correction coordinates. By iterating through the screen-normalized coordinates of all geometric vertices and performing inverse affine compensation, we obtain the set of boundary coordinates of all geometric vertices mapped to the screen's two-dimensional space, as well as the discrete vertex depth values ​​corresponding to the geometric vertices under the camera's view frustum, thus forming the geometric skeleton for subsequent pixel-level rendering. Based on the geometric skeleton, primitive assembly and scanning transformation are performed according to the boundary coordinate set to determine the two-dimensional pixel area covered by the UI on the screen. For any screen pixel to be rendered within the two-dimensional pixel area, linear interpolation calculation is performed on the discrete vertex depth value using the proportion of the centroid coordinates of the pixel inside the polygon to obtain the virtual interpolation depth corresponding to the pixel. Based on the original depth map of the three-dimensional pose coordinates and the dense three-dimensional point cloud, a real scene depth map of the current frame is generated. The pixel-level depth difference between the virtual interpolation depth and the real scene depth map is calculated. The pixel-level depth difference is substituted into the transparency decay function to calculate the depth cancellation transparency coefficient corresponding to each pixel in the two-dimensional pixel region, thereby realizing the soft cancellation of depth conflicts in the virtual UI. The gaze unit vector is projected onto the screen of the eye-tracking digital terminal to obtain the two-dimensional gaze point coordinates. For the two-dimensional pixel area, the Euclidean distance from any pixel to the gaze point is calculated. Based on the foveal vision mechanism, the dynamic downsampling factor is calculated. Using the GPU of the eye-tracking digital terminal, the gaze center area of ​​the effective UI pixel area is rendered at 1:1 full resolution according to the dynamic downsampling factor, while the edge leading-out area of ​​the effective UI pixel area is rendered at low resolution. According to the transparency attenuation parameter, the material background color of the expert interpretation text, seal interpretation text and simplified / traditional text in the effective UI pixel area is weighted and fused with the physical background color of the eye-tracking digital terminal to generate a virtual and real fusion image of calligraphy and painting, which is then displayed.

10. A reality-based calligraphy and painting virtual-real fusion display device, used to implement the calligraphy and painting virtual-real fusion display method as described in any one of claims 1-9, characterized in that, The device includes: The spatial perception unit is used to scan the display of calligraphy and painting appreciation works on an augmented reality eye-tracking digital terminal, and to construct a dense geometric layer and a sparse tracking layer of a two-layer data architecture. The dense geometric layer includes a dense three-dimensional point cloud, and the sparse tracking layer includes a sparse spatial map. The information anchor point generation unit is used to extract information anchor points of specific elements in the display of calligraphy and painting appreciation works, and bind them with the corresponding digital information data packets to generate a set of information anchor points including three-dimensional polygon boundaries and digital information data packets. The eye-tracking unit is used to perform eye tracking and gaze point calculation on the user based on dense 3D point cloud and sparse spatial map, to obtain accurate gaze point coordinates, and to perform hit determination and activation based on information anchor point set, to obtain target digital information data packet and target 3D polygon boundary of the target anchor point hit by the user's gaze. The optimized rendering unit is used to construct the substrate and potential field environment based on dense 3D point clouds, according to the precise coordinates of the viewpoint and the boundary of the target 3D polygon, and to start a multi-objective enzyme-based optimization algorithm to solve the user's 3D pose coordinates. The virtual-real fusion display unit is used to dynamically overlay and render target digital information data packets onto the display screen of calligraphy and painting appreciation works based on three-dimensional pose coordinates, so as to realize the virtual-real fusion display of calligraphy and painting.

Images