An augmented reality-based cultural and creative product design display method and system

Abstract

The application provides an augmented reality-based cultural and creative product design display method and system, collects real-time images and inertial measurement unit original data of a cultural and creative showcase, extracts showcase frame line features and cultural and creative prototype exhibit silhouette features, matches the features with a pre-constructed showcase semantic map to determine an initial pose, continuously tracks inter-frame pixel displacement during movement to update a real-time pose, extracts a visible area of the exhibit silhouette features, performs local registration on the visible area and a projection of a three-dimensional model edge in the semantic map at a previous effective pose, recovers the pose with an inertial measurement unit pre-integration predicted pose as an initial value, verifies whether the recovery tracking and real-time pose updating are passed, and if not, triggers global repositioning, and renders a cultural and creative product virtual model to a corresponding exhibit silhouette position in the showcase according to the real-time pose to generate a display picture. By using the scheme, the pose can be corrected through a visible local area of a cultural and creative prototype exhibit silhouette to maintain stable display.

Description

Technical Field

[0001] This application relates to the field of product display technology, and more specifically, to a method and system for designing and displaying cultural and creative products based on augmented reality. Background Technology

[0002] Product demonstration refers to the construction of a digital expression system that deconstructs and reorganizes product information at multiple levels and conveys it to the viewer in a highly perceptible way. In the traditional model, it is often limited to flat brochures, videos from a fixed perspective, or physical prototypes, and the viewer can only passively receive selected and fragmented information. With the popularization of smart terminals and 3D vision technology, product demonstration has evolved into a continuous and dynamic experience process. It refers to a series of digital processing processes, such as high-precision image acquisition, calculation of material and light and shadow physical properties, 3D model reconstruction or parametric design, and visual programming of interactive logic.

[0003] In the existing process of designing and displaying cultural and creative products based on augmented reality, traditional methods rely on general point features for pose tracking. However, in the context of museum display cases, glass reflection, sparse textures, and partial occlusion can easily lead to feature loss and mismatch, causing tracking to be interrupted frequently. Especially when the border features of the display case are obscured, the remaining partial silhouette of the cultural and creative product lacks regular geometric attributes. Traditional methods cannot effectively register this unstructured outline with the pre-labeled 3D spatial information in the semantic map of the display case, nor do they have a reliable verification mechanism for restoring the pose, often causing the virtual model to jump or drift. For example, when a visitor obscures part of the display case border, the system can only observe the incomplete area of ​​the silhouette of the cultural and creative product. Traditional algorithms cannot constrain the registration and consistency verification of this visible part with the projection outline of the 3D model of the cultural and creative product under the previous effective pose, which easily leads to the partial silhouette being incorrectly matched to other parts of the model, causing the superimposed virtual model of the cultural and creative product to deviate from the correct position instantly. Therefore, how to perform pose correction through the visible local area of ​​the silhouette of the cultural and creative product to maintain display stability has become a difficult problem for the industry. Summary of the Invention

[0004] This application provides a method and system for designing and displaying cultural and creative products based on augmented reality, which can perform pose correction through the visible local area of ​​the silhouette of the cultural and creative prototype exhibit to maintain display stability.

[0005] Firstly, this application provides a method for designing and displaying cultural and creative products based on augmented reality, comprising the following steps: Real-time images of the cultural and creative product display cases and raw data from the inertial measurement unit of the user equipment are collected, and the features of the display case border lines and the silhouette features of the cultural and creative product prototypes in the real-time images are extracted. The display case border features and the silhouette features of the cultural and creative prototype exhibits are matched with the pre-constructed semantic map of the display case to determine the initial pose of the user device relative to the display case. During the process of the user moving the device, the inter-frame pixel displacement of the display case border line features and the silhouette features of the cultural and creative prototype exhibit is continuously tracked. Combined with the pre-integrated data of the inertial measurement unit, the real-time pose of the user device is updated from the initial pose. When the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit is extracted. The visible area is locally registered with the projection of the edge point set of the 3D model of the cultural and creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is restored and verified with the inertial pre-integration predicted pose as the initial value. If the verification is successful, tracking is resumed and the real-time pose is updated. Otherwise, the global relocalization process is triggered. Based on the real-time pose, the virtual model of the cultural and creative product is rendered onto the spatial position corresponding to the silhouette feature of the prototype cultural and creative exhibit within the display case, generating a display screen.

[0006] In some embodiments, matching the display case border features and the silhouette features of the cultural and creative prototype exhibit with a pre-built semantic map of the display case to determine the initial pose of the user device relative to the display case specifically includes: The features of the display case border line are matched with the 3D border information of the display case stored in the pre-constructed semantic map of the display case to establish the correspondence between the 2D line segments in the image and the 3D border in the world coordinate system. The initial pose of the user equipment relative to the display case is calculated based on the aforementioned correspondence. The silhouette features of the cultural and creative prototype exhibit are matched with the projection contour of the 3D model of the cultural and creative prototype exhibit in the semantic map of the display case under the initial pose, and the initial pose is corrected according to the matching result to determine the initial pose of the user device relative to the display case.

[0007] In some embodiments, during the process of a user moving a mobile device, continuously tracking the inter-frame pixel displacement of the display case border features and the silhouette features of the cultural and creative prototype exhibit, and combining the pre-integrated data from the inertial measurement unit, updating the real-time pose of the user device starting from the initial pose specifically includes: During the process of the user's mobile device, a set of feature points for tracking is extracted from the display case border line features and the silhouette features of the cultural and creative prototype exhibits in the current frame; Based on the feature point set of the previous frame, the pixel coordinates of each feature point in the feature point set are tracked in the current frame to obtain the inter-frame pixel displacement of the feature point set. The raw data from the inertial measurement unit is pre-integrated to generate a predicted value of the relative pose increment between adjacent frames. Based on the inter-frame pixel displacement and the real-time pose of the previous frame, the corresponding 3D points in the semantic map of the display case are projected onto the current frame to establish the correspondence between the 3D points and the 2D tracking points. Using the inertial pre-integration relative pose increment prediction value as the initial value, the real-time pose of the current frame is calculated using the correspondence between the three-dimensional points and the two-dimensional tracking points, and the real-time pose of the user equipment is updated.

[0008] In some embodiments, when the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, extracting the visible area of ​​the silhouette features of the cultural and creative prototype exhibit specifically includes: When the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, the foreground mask is obtained from the silhouette features of the cultural and creative prototype exhibit in the current frame to determine the pixel area occupied by the cultural and creative prototype exhibit in the image. Extract the outer boundary contour of the silhouette features of the cultural and creative prototype exhibit within the pixel area to obtain the set of visible contour points; Uniform sampling is performed on the set of visible contour points to extract the visible area of ​​the silhouette features of the cultural and creative prototype exhibit.

[0009] In some embodiments, the visible area is locally registered with the projection of the edge point set of the 3D model of the creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is recovered and verified using the inertial pre-integration predicted pose as the initial value. If the verification is successful, tracking is resumed and the real-time pose is updated; otherwise, a global relocalization process is triggered as the real-time pose. Specifically, this includes: The raw data from the inertial measurement unit during the tracking interruption period is pre-integrated to generate the inertial pre-integrated predicted pose for the current frame. The sampling points in the visible area are used as the set of points to be registered, and the projection of the edge point set of the three-dimensional model of the original Chinese prototype exhibit in the semantic map of the display case under the inertial pre-integration predicted pose is used as the target point set. Perform iterative nearest-point algorithm for local registration to obtain the pose to be verified in the current frame; Calculate the reprojection error of the prototype exhibit of the pose to be verified, and verify the pose to be verified based on the reprojection error; When the verification is successful, the pose to be verified is taken as the current real-time pose and tracking is resumed, and the real-time pose of the user device is updated. If the verification fails, extract all visible display case border lines and exhibit silhouette features in the current frame, perform global feature matching with the display case semantic map, and re-solve the initial pose.

[0010] In some embodiments, rendering the virtual model of the cultural and creative product onto the spatial position corresponding to the silhouette feature of the prototype cultural and creative exhibit within the display case, based on the real-time pose, to generate the display screen specifically includes: Based on the real-time pose, determine the spatial position of the 3D model of the cultural and creative prototype exhibit corresponding to the silhouette feature of the cultural and creative prototype exhibit in the semantic map of the display case; Based on the spatial position of the three-dimensional model of the cultural and creative prototype exhibit, the placement position of the virtual model of the cultural and creative product is determined, and the model transformation matrix of the virtual model of the cultural and creative product is constructed. A projection matrix is ​​constructed using the real-time pose and camera intrinsic parameter matrix to project the virtual model of the cultural and creative product from the three-dimensional world coordinate system to the two-dimensional image plane, generating a rendering layer. The rendered layer is overlaid and combined with the real-time image of the current frame to generate the display screen.

[0011] In some embodiments, the semantic map of the display case includes three-dimensional spatial information of the display case and the cultural and creative prototype exhibits inside, as well as the relative topological position information of the exhibits.

[0012] Secondly, this application provides an augmented reality-based design and display system for cultural and creative products, including: The acquisition module is used to acquire real-time images of the cultural and creative display case and raw data from the inertial measurement unit of the user equipment, and to extract the display case border features and the silhouette features of the cultural and creative prototype exhibits from the real-time images. The processing module is used to match the border features of the display case and the silhouette features of the cultural and creative prototype exhibit with the pre-built semantic map of the display case to determine the initial pose of the user device relative to the display case. The processing module is also used to continuously track the inter-frame pixel displacement of the display case border line features and the silhouette features of the cultural and creative prototype exhibits during the user's mobile device operation, and update the real-time pose of the user device with the initial pose as the starting point by combining the pre-integrated data of the inertial measurement unit. The processing module is also used to extract the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit when the number of effective border line features is lower than a preset threshold, causing the tracking to be interrupted. The visible area is then locally registered with the projection of the edge point set of the three-dimensional model of the cultural and creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is restored and verified using the inertial pre-integration predicted pose as the initial value. If the verification is successful, the tracking is resumed and the real-time pose is updated. Otherwise, the global relocalization process is triggered. The execution module is used to render the virtual model of the cultural and creative product onto the spatial position corresponding to the silhouette feature of the cultural and creative prototype exhibit within the display case, based on the real-time pose, and generate a display screen.

[0013] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described augmented reality-based cultural and creative product design and display method.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described augmented reality-based cultural and creative product design and display method.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The augmented reality-based method and system for designing and displaying cultural and creative products provided in this application firstly includes a semantic map of the display case containing three-dimensional spatial information of the display case and the cultural and creative prototype exhibits inside. This process elevates the cultural and creative prototype exhibits from two-dimensional silhouettes dependent on the viewing angle to a spatial structure prior independent of the viewing angle, thus providing a unique and clear three-dimensional reference benchmark for pose correction based solely on locally visible silhouettes. This allows fragmented visible outlines to be constrained to known geometry, rather than being isolated features lacking spatial belonging. Subsequently, the silhouettes of the cultural and creative prototype exhibits in the visible area are locally registered with the projections of the edge point sets of the three-dimensional model of the cultural and creative prototype exhibits under the previous effective pose. This registration operation uses the pre-integration predicted pose of the inertial measurement unit as the initial value for iteration, constraining visual feature matching within the neighborhood of the actual motion trajectory. Simultaneously, relying on the three-dimensional spatial information, the originally ambiguous two-dimensional to two-dimensional matching is transformed into a spatial consistency check constrained by the projection of the three-dimensional model. By comparing the theoretical outline projected from the edge point set of the complete geometric model with the local silhouette, a spatially interpretable matching target is provided for pose correction, avoiding... To address the issue of mismatched silhouettes to non-corresponding areas, and to meet the final display requirement of rendering the virtual model of the cultural and creative product onto the corresponding spatial position of the silhouette of the prototype exhibit within the display case, the pose correction must meet the rigid spatial condition of accurately superimposing the virtual model onto the real 3D coordinates of the prototype exhibit. This process elevates simple contour matching to a spatial positioning task oriented towards the rendering target, providing quantifiable verification evidence for the recovered pose and making it possible to determine the reliability of the pose through methods such as reprojection errors. Furthermore, when pose verification fails, a global repositioning process is triggered instead of relying on local information to continuously attempt recovery, avoiding the cumulative error caused by repeated iterations in the neighborhood of the incorrect pose. Simultaneously, the continuous acquisition of raw data from the inertial measurement unit and the re-detection of the display case's border line features are integrated to redetermine the global pose of the user device through multi-source information collaboration, ensuring that the virtual model can be stably anchored on the correct 3D coordinates of the prototype exhibit, avoiding model jumps or drifts due to mismatches. In summary, this solution can perform pose correction through the visible local area of ​​the silhouette of the prototype exhibit to maintain display stability. Attached Figure Description

[0016] Figure 1This is a flowchart illustrating a method for designing and displaying cultural and creative products based on augmented reality, according to some embodiments of this application. Figure 2 This is a flowchart illustrating the determination of the initial pose according to some embodiments of this application; Figure 3 This is a flowchart illustrating the process of determining a set of feature points according to some embodiments of this application; Figure 4 This is a structural schematic diagram of an augmented reality-based cultural and creative product design and display system according to some embodiments of this application; Figure 5 This is an internal structural diagram of a computer device that implements an augmented reality-based method for designing and displaying cultural and creative products, according to some embodiments of this application. Detailed Implementation

[0017] To better understand the technical solutions in this embodiment, the technical solutions in this embodiment will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0018] refer to Figure 1 The figure is a flowchart illustrating a method for designing and displaying cultural and creative products based on augmented reality, according to some embodiments of this application. This method mainly includes the following steps: In step 101, real-time images of the cultural and creative display case and raw data from the inertial measurement unit of the user equipment are collected, and the border line features of the display case and the silhouette features of the cultural and creative prototype exhibits are extracted from the real-time images.

[0019] In practice, the monocular camera on the user's mobile device is first calibrated to obtain its intrinsic parameters and distortion coefficients. Based on this, distortion correction processing is performed on each real-time image frame to eliminate geometric distortion caused by wide-angle or lens defects. Simultaneously, histogram equalization is used to stretch the grayscale distribution range of the image, enhancing the contrast of the backlit areas of the display case in low-light environments. After preprocessing, feature extraction of the display case border lines and silhouette features of the cultural and creative prototype exhibits are performed in parallel. For the display case border line features, a line segment detection algorithm is used to quickly detect straight line segments that conform to the gradient direction continuity from the image, setting the maximum... The initial set of line segments is obtained by setting the length of the detected line segments to one-tenth of the shorter side of the image and the tolerance for directional differences to fifteen degrees. Then, using the prior geometric constraint that the display case border should be a rectangular structure, clusters of line segments that are parallel and orthogonal to each other are selected from the initial set of line segments. Based on this, if the number of line segment clusters is ≥4 and can form a complete rectangle, the actual physical aspect ratio of the display case stored in the pre-constructed display case semantic map is introduced to calculate the projected aspect ratio of the above rectangular candidate regions in the image. Only rectangular boundaries with a deviation of less than 20% are retained as valid display case border line features. If the number of line segment clusters is 2-3... If a line meets the directional constraints of the display case border (i.e., horizontal / vertical deviation ≤ 15°), it is directly marked as a valid display case border line feature. For the silhouette features of cultural and creative prototype exhibits, a background model composed of multiple Gaussian distribution weighted combinations can be constructed for each pixel in the image to depict the appearance of relatively static background objects such as the display case back panel and lining cloth that change slowly over time. After the current frame is subtracted from the background model, a coarse segmentation region containing foreground objects such as cultural and creative prototype exhibits is obtained. Then, morphological erosion operations are used to remove isolated noise points, and morphological dilation operations are used to fill the small holes inside the region caused by reflection or texture. Finally, a clear silhouette feature of cultural and creative prototype exhibits is obtained. Simultaneously, acceleration data and angular velocity data output by the user equipment's built-in inertial measurement unit are collected. The sampling frequency can be set to 200Hz for example. Zero bias correction and noise filtering are performed on the raw data to obtain the raw data of the inertial measurement unit.

[0020] It should be noted that the cultural and creative product display case described in this application displays cultural relics, and the user's device displays cultural and creative products next to the corresponding cultural relics in the display case; the display case border line feature refers to the set of straight line segments extracted from the two-dimensional image that correspond to the four sides of the rectangular outline of the cultural and creative product display case. Under perspective projection, it is represented as a set of two-dimensional lines with specific geometric relationships. Its function is to provide a macroscopic and stable geometric reference for camera pose calculation. Compared with scattered feature points, line features have stronger resistance to local occlusion and glass reflection; the cultural and creative prototype exhibit silhouette feature refers to the foreground mask and its outer boundary outline that occupy a certain continuous pixel area in the image. Semantically, it corresponds to the two-dimensional projection shape of the real cultural and creative prototype exhibit; the mixed Gaussian background modeling is an adaptive background subtraction method that adapts to the gradual change of lighting and dynamic background. It does not assume that the background pixel value is a fixed constant, but models its historical value distribution as a weighted mixture of multiple Gaussian components, so that it can still stably identify the foreground cultural and creative prototype exhibit when the lighting changes slowly or the lining of the display case moves slightly; the line segment detection algorithm is a straight line extraction method based on the local consistency of gradient magnitude and direction; this application does not limit the specific implementation variants of the above algorithms. Any method that can reliably extract the features of the display case border line and the silhouette features of the cultural and creative prototype exhibit can be applied to this solution.

[0021] In step 102, the features of the display case border line and the silhouette features of the cultural and creative prototype exhibit are matched with the pre-constructed semantic map of the display case to determine the initial pose of the user device relative to the display case.

[0022] In this embodiment, the semantic map of the display case includes three-dimensional spatial information of the display case and the prototype cultural and creative exhibits inside, as well as the relative topological position information of the exhibits; in some embodiments, reference is made to... Figure 2 As shown in the figure, this is a flowchart illustrating the process of determining the initial pose in some embodiments of this application. The initial pose of the user device relative to the display case can be determined by matching the features of the display case border and the silhouette features of the cultural and creative prototype exhibit with a pre-built semantic map of the display case. This can be achieved through the following steps: First, in step 1021, the features of the display case border line are matched with the three-dimensional border information of the display case stored in the pre-constructed display case semantic map to establish the correspondence between the two-dimensional line segments in the image and the three-dimensional border in the world coordinate system. Then, in step 1022, the preliminary pose of the user equipment relative to the display case is calculated based on the correspondence; Finally, in step 1023, the silhouette features of the cultural and creative prototype exhibit are matched with the projection contour of the three-dimensional model of the cultural and creative prototype exhibit in the semantic map of the display case under the initial pose, and the initial pose is corrected according to the matching result to determine the initial pose of the user equipment relative to the display case.

[0023] In specific implementation, the feature of the display case border line is matched with the 3D border information of the display case stored in the pre-constructed semantic map of the display case to establish the correspondence between the 2D line segments in the image and the 3D border in the world coordinate system. This can be achieved in the following way: For example, four 2D line segments constituting the rectangular boundary are obtained from the effective display case border line features, and the endpoint coordinates, line direction, and line length of each line segment in the image are extracted as geometric description attributes; at the same time, the corner world coordinates of the four spatial edges of the 3D border of the display case are read from the semantic map of the display case, and the four 3D edges are arranged in a clockwise or counterclockwise order according to the topological connection relationship of the 3D border; then, the geometric description attributes of the 2D line segments are associated and matched with the 3D edges, based on the concavity and convexity of the rectangle vertices. The degree of consistency and the closeness of the angle between adjacent sides to a right angle are used as matching constraints. Each line segment of the two-dimensional rectangle boundary is sequentially assigned to the edge of the three-dimensional border that is closest to it in direction and has the best relative position, thereby establishing a one-to-one correspondence between the two-dimensional line segments in the image and the three-dimensional border in the world coordinate system. The association matching process uses the actual physical aspect ratio of the display case stored in the display case semantic map as a prior constraint, and prioritizes matching the candidate line segment combination that is most consistent with the actual aspect ratio of the display case, thereby reducing the probability of mismatched line segments due to local occlusion or reflection. When there are multiple cultural and creative prototype exhibits in the display case, the relative topological position information of the exhibits stored in the display case semantic map is used to sort and match the silhouette features of multiple exhibits, eliminating the interference of exhibits with similar shapes.

[0024] It should be noted that the correspondence described in this application refers to the matching relationship established between the two-dimensional display case border line segments extracted from the image and the three-dimensional border edges of the display case in the world coordinate system. Its function is to associate and bind the geometric observations in the two-dimensional image with the pre-constructed three-dimensional spatial prior information.

[0025] In specific implementation, the preliminary pose of the user equipment relative to the display case can be calculated based on the aforementioned correspondence in the following manner: For example, the correspondence between the two-dimensional line segments in the image and the three-dimensional bounding box edges in the world coordinate system is input into the perspective multi-line pose solver. The spatial straight line equations of the three-dimensional bounding box edges in the world coordinate system and the straight line equations of the corresponding two-dimensional line segments on the image plane are used as inputs. The rotation matrix and translation vector of the user equipment camera coordinate system relative to the world coordinate system are solved by minimizing the geometric distance between the projection line of the three-dimensional straight line on the image plane and the corresponding two-dimensional line segment. The constraint of this optimization process is that the three-dimensional straight line should fall on the straight line where the corresponding two-dimensional line segment is located after being projected by the camera intrinsic parameter matrix. The calculated rotation matrix and translation vector are the preliminary pose of the user equipment relative to the display case. The perspective multi-line pose solver uses an iterative optimization method to gradually approach the optimal solution. The initial iteration value can be roughly estimated by the correspondence between the four corner points of the display case bounding box in two-dimensional and three-dimensional coordinates using a direct linear transformation method. This application does not limit this.

[0026] It should be noted that the preliminary pose mentioned in this application refers to the user equipment pose estimate obtained by initially calculating the correspondence between two-dimensional line segments and three-dimensional border edges after matching the display case border line features with the display case semantic map.

[0027] In specific implementation, the silhouette features of the cultural and creative prototype exhibit are matched with the projected contour of the 3D model of the cultural and creative prototype exhibit in the semantic map of the exhibition case under the initial pose. The initial pose is then corrected based on the matching result. The initial pose of the user device relative to the exhibition case can be determined in the following way: First, the 3D model of the cultural and creative prototype exhibit stored in the semantic map of the exhibition case is projected onto the image plane based on the initial pose, generating a binary projection contour map with the same size as the current image and containing only the outline area of ​​the cultural and creative prototype exhibit. Then, the outer boundary contour of the silhouette features of the cultural and creative prototype exhibit extracted in step one is matched with the binary projection contour map. The geometric similarity between the silhouette feature contour and the projection contour is calculated. The measurement of geometric similarity can comprehensively consider the global shape consistency of the contour and the local edge overlap. When the geometric similarity meets the preset matching conditions, it indicates that the initial pose is sufficient. The initial pose is precisely used as the user device's initial pose relative to the display case. In this embodiment, when the geometric similarity does not meet the preset matching conditions, a small perturbation is applied to the Lie algebra corresponding to the rotation matrix in the neighborhood space of the initial pose to generate multiple candidate rotation matrices to ensure orthogonality. At the same time, a small perturbation is applied to each component of the translation vector to generate multiple candidate poses. The candidate pose that has the highest geometric similarity between the silhouette feature contour and the projected contour of the cultural and creative prototype exhibit is selected to replace the initial pose, and this candidate pose is used as the corrected initial pose. The geometric similarity threshold in the preset matching conditions is set based on the complexity and image resolution of the three-dimensional model of the cultural and creative prototype exhibit. The more complex the three-dimensional model of the cultural and creative prototype exhibit and the higher the image resolution, the more stringent the geometric similarity threshold can be set, and vice versa. This application does not limit the specific value of the threshold.

[0028] It should be noted that the initial pose mentioned in this application refers to the user equipment's initial pose that is finally confirmed after contour matching and pose correction of the silhouette features of the cultural and creative prototype exhibit and the projection outline of the three-dimensional model of the cultural and creative prototype exhibit, based on the preliminary pose. Its function is to serve as the starting point of the entire tracking process.

[0029] In step 103, during the process of the user moving the device, the inter-frame pixel displacement of the display case border line features and the silhouette features of the cultural and creative prototype exhibit is continuously tracked. Combined with the pre-integrated data of the inertial measurement unit, the real-time pose of the user device is updated starting from the initial pose.

[0030] In some embodiments, during the process of a user moving a mobile device, continuously tracking the inter-frame pixel displacement of the display case border features and the silhouette features of the cultural and creative prototype exhibit, and combining the pre-integrated data from the inertial measurement unit, updating the real-time pose of the user device starting from the initial pose can be achieved through the following steps: During the process of the user's mobile device, a set of feature points for tracking is extracted from the display case border line features and the silhouette features of the cultural and creative prototype exhibits in the current frame; Based on the feature point set of the previous frame, the pixel coordinates of each feature point in the feature point set are tracked in the current frame to obtain the inter-frame pixel displacement of the feature point set. The raw data from the inertial measurement unit is pre-integrated to generate a predicted value of the relative pose increment between adjacent frames. Based on the inter-frame pixel displacement and the real-time pose of the previous frame, the corresponding 3D points in the semantic map of the display case are projected onto the current frame to establish the correspondence between the 3D points and the 2D tracking points. Using the inertial pre-integration relative pose increment prediction value as the initial value, the real-time pose of the current frame is calculated using the correspondence between the three-dimensional points and the two-dimensional tracking points, and the real-time pose of the user equipment is updated.

[0031] For specific implementation, refer to Figure 3 As shown in the figure, this is a flowchart illustrating the process of determining a feature point set according to some embodiments of this application. During the process of a user moving a mobile device, the feature point set for tracking can be extracted from the display case border line features and the silhouette features of the cultural and creative prototype exhibit in the current frame in the following manner: For example, obtain the valid display case border line features extracted in the current frame; for each complete border line segment, take its two endpoints and midpoint as border-type tracking points; for occluded incomplete border line segments, take its two endpoints and midpoint as border-type tracking points; at the same time, for the silhouette features of the cultural and creative prototype exhibit extracted in the current frame, extract contour points on its outer boundary contour in an equally spaced sampling manner as silhouette-type tracking points; merge the border-type tracking points and the silhouette-type tracking points to form a feature point set for inter-frame tracking.

[0032] It should be noted that the feature point set mentioned in this application refers to a set of discrete two-dimensional points extracted from the display case border line features and the silhouette features of the cultural and creative prototype exhibits in the current frame for inter-frame tracking.

[0033] In specific implementation, based on the feature point set of the previous frame, the pixel coordinates of each feature point in the feature point set are tracked in the current frame to obtain the inter-frame pixel displacement of the feature point set. This can be achieved in the following way: taking each feature point in the feature point set as the center, a neighborhood image patch in the previous frame image is taken as the appearance description template of the feature point; in the current frame image, taking the pixel coordinates of the feature point in the previous frame as the center and within a preset search window, the best matching position of the feature point in the current frame is searched by minimizing the sum of squared gray-level differences between the appearance description template and the current frame image patch, thus obtaining the pixel coordinates of each feature point in the current frame; the pixel coordinates of each feature point in the current frame are then... Subtracting the pixel coordinates of the feature point from the previous frame yields the inter-frame pixel displacement vector of each feature point in the feature point set. During tracking, the normalized cross-correlation coefficient between the appearance description template of each feature point and the best matching image block in the current frame is calculated. Low-quality tracking points with cross-correlation coefficients lower than a preset quality threshold are eliminated to ensure the reliability of the inter-frame pixel displacement vector. The preset quality threshold is set based on the richness of the texture and the amplitude of illumination variation in the current scene. The richer the scene texture and the smaller the illumination variation, the higher the preset quality threshold can be set to retain more accurate tracking points. Conversely, it should be appropriately lowered to retain a sufficient number of tracking points. This application does not limit the specific value of the threshold.

[0034] It should be noted that the inter-frame pixel displacement mentioned in this application refers to the two-dimensional displacement vector between the pixel coordinates of the same feature point in the previous frame and the pixel coordinates in the current frame. Its function is to quantify the changes in image content caused by the movement of the user device.

[0035] In specific implementation, the pre-integration processing of the raw data of the inertial measurement unit to generate the predicted relative pose increment between adjacent frames can be achieved in the following way: for example, acquiring all raw data of the inertial measurement unit between the acquisition time of the previous frame image and the acquisition time of the current frame image; performing numerical integration on the acceleration data and angular velocity data respectively to calculate the rotation increment and translation increment of the device during this time period; converting the rotation increment and translation increment into the relative pose increment in the camera coordinate system, which is the predicted relative pose increment between adjacent frames; during the pre-integration process, the median integration method is used to improve the integration accuracy, and the zero bias of the inertial measurement unit is estimated and compensated online.

[0036] In specific implementation, based on the inter-frame pixel displacement and the real-time pose of the previous frame, the corresponding 3D points in the display case semantic map are projected onto the current frame. The correspondence between 3D points and 2D tracking points can be established in the following ways: for example, using the real-time pose of the previous frame, the 3D spatial points in the display case semantic map corresponding to each feature point in the feature point set are projected onto the image plane of the current frame to obtain the predicted projection position of each 3D spatial point in the current frame; the predicted projection position of each 3D spatial point is associated with the actual pixel coordinates of the corresponding feature points in the current frame obtained based on inter-frame pixel displacement tracking, and a one-to-one correspondence between the 3D spatial points and the 2D tracking points of the current frame is established one by one; for silhouette-type tracking points that lack corresponding 3D spatial points in the display case semantic map, observation data of more than 3 consecutive frames are accumulated, and the baseline formed by the camera motion between multiple frames is used for triangulation to reconstruct the coordinates of the corresponding 3D spatial points, and the reconstructed 3D spatial points are added to the display case semantic map.

[0037] It should be noted that the correspondence between the three-dimensional points and the two-dimensional tracking points mentioned in this application refers to the matching relationship established between the known three-dimensional spatial points in the display case semantic map and the two-dimensional feature points tracked in the current frame image. Its function is to associate the pre-constructed three-dimensional scene information with real-time image observation.

[0038] In specific implementation, the inertial pre-integration relative pose increment prediction value is used as the initial value. The real-time pose of the current frame is calculated using the correspondence between the three-dimensional points and the two-dimensional tracking points. The real-time pose of the user equipment can be updated in the following way, for example: starting from the initial pose, iteratively calculate frame by frame, multiplying the real-time pose of the previous frame with the inertial pre-integration relative pose increment prediction value to obtain the initial value for iteratively solving the pose of the current frame; using the set of correspondences between the three-dimensional points and the two-dimensional tracking points as input, a perspective multi-point pose calculation method is adopted, with the optimization objective being to minimize the reprojection error between the pixel position of the three-dimensional space point after projection through the camera intrinsic parameter matrix and the pose to be calculated in the current frame and the actual pixel coordinates of the corresponding two-dimensional tracking point, and iteratively solving to obtain the current pose. The rotation matrix and translation vector of the previous frame are used as the real-time pose of the current frame, replacing the real-time pose of the previous frame, thus updating the real-time pose of the user equipment. During the iterative solution process, if the reprojection error of a certain pair of correspondences exceeds a preset error tolerance range, the correspondence is marked as an abnormal match and removed. The remaining valid correspondences are then recalculated to improve the accuracy of the pose update. The preset error tolerance range is set based on image resolution and feature point extraction accuracy. Higher image resolution and higher feature point extraction accuracy allow for a more stringent preset error tolerance range, while lower resolution and lower feature point extraction accuracy allow for a more relaxed range. This application does not limit the specific value of this tolerance range.

[0039] It should be noted that the real-time pose of the current frame mentioned in this application refers to the latest pose obtained by solving and updating the spatial position and orientation of the device corresponding to the current frame image using inter-frame pixel displacement and pose information of the previous frame during the movement of the user device. Its function is to reflect the changes in the user's viewpoint relative to the display case in real time.

[0040] In step 104, when the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit is extracted, and the visible area is locally registered with the projection of the edge point set of the 3D model of the cultural and creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is recovered and verified with the inertial pre-integration predicted pose as the initial value. If the verification is successful, tracking is resumed and the real-time pose is updated; otherwise, the global relocalization process is triggered.

[0041] In some embodiments, when the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, the visible area of ​​the silhouette features of the cultural and creative prototype exhibit can be extracted using the following steps: When the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, the foreground mask is obtained from the silhouette features of the cultural and creative prototype exhibit in the current frame to determine the pixel area occupied by the cultural and creative prototype exhibit in the image. Extract the outer boundary contour of the silhouette features of the cultural and creative prototype exhibit within the pixel area to obtain the set of visible contour points; Uniform sampling is performed on the set of visible contour points to extract the visible area of ​​the silhouette features of the cultural and creative prototype exhibit.

[0042] In specific implementation, when the number of effective border line features is lower than a preset threshold, causing tracking interruption, a foreground mask is obtained from the silhouette features of the cultural and creative prototype exhibit in the current frame. The determination of the pixel area occupied by the cultural and creative prototype exhibit in the image can be achieved in the following way: For example, the foreground mask contained in the silhouette features of the cultural and creative prototype exhibit already generated in the current frame is obtained. This foreground mask is a binary image with the same size as the current frame image, in which pixels belonging to the cultural and creative prototype exhibit area are marked as foreground and the remaining pixels are marked as background. Connectivity analysis is performed on the foreground mask, and the connected component with the largest area is extracted as the main body area of ​​the cultural and creative prototype exhibit. The pixel coordinate range covered by the bounding rectangle of the main body area of ​​the cultural and creative prototype exhibit is determined as the pixel area occupied by the cultural and creative prototype exhibit in the image, thereby filtering out the interference of sporadic false detection small areas caused by reflection or shadow in the foreground mask on subsequent contour extraction. The preset threshold for the number of effective border line features is set to 2, that is, when the number of detected effective display cabinet border line segments is less than 2, it is determined that tracking is interrupted.

[0043] In specific implementation, the outer boundary contour of the silhouette feature of the cultural and creative prototype exhibit within the pixel area can be extracted to obtain the visible contour point set in the following way: For example, within the pixel area occupied by the cultural and creative prototype exhibit in the image, a boundary tracking algorithm is executed on the foreground mask, and the boundary pixels between the main body area of ​​the cultural and creative prototype exhibit and the background are traversed in a clockwise or counterclockwise direction to extract all contour points that constitute the outer boundary of the cultural and creative prototype exhibit; all the extracted contour points are arranged into an ordered point sequence according to the traversal order, which is the visible contour point set; wherein, the boundary tracking algorithm starts from any foreground pixel on the edge of the main body area of ​​the cultural and creative prototype exhibit, searches for the next boundary pixel point by point along the neighborhood direction until returning to the starting point, ensuring that the extracted contour forms a closed and non-self-intersecting continuous boundary line as the visible contour point set.

[0044] It should be noted that the set of visible outline points mentioned in this application refers to the set of pixels that constitute the outer boundary of the visible part of the cultural and creative prototype exhibit, extracted from the unobstructed area of ​​the foreground mask of the silhouette feature of the cultural and creative prototype exhibit when the frame of the display case is obscured and the tracking is interrupted.

[0045] In specific implementation, uniform sampling of the visible contour point set and extraction of the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit can be achieved in the following way: For example, calculate the total contour arc length of the visible contour point set, and select a number of sampling points from the visible contour point set at equal arc length intervals along the contour arc length direction; connect the selected sampling points in their original order on the contour to form a polygonal area as the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit; the setting of the equal arc length interval is based on the contour complexity of the silhouette feature of the cultural and creative prototype exhibit and the computing power of the user device. The more complex the contour and the stronger the computing power, the smaller the sampling interval can be set to retain richer contour details for subsequent local registration, and vice versa, the sampling interval can be appropriately increased to ensure the real-time processing. This application does not limit the specific value of the sampling interval.

[0046] It should be noted that the visible region mentioned in this application refers to the polygonal region formed by connecting the sampled points after uniformly sampling the set of visible contour points. This region represents the geometric shape of the visible part of the silhouette of the Chinese creative prototype exhibit in the current frame and serves as the source data point set for local registration.

[0047] In some embodiments, the visible area is locally registered with the projection of the edge point set of the 3D model of the creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is recovered and verified using the inertial pre-integration predicted pose as the initial value. If the verification is successful, tracking is resumed and the real-time pose is updated. Otherwise, the global relocalization process is triggered, which can be implemented using the following steps: The raw data from the inertial measurement unit during the tracking interruption period is pre-integrated to generate the inertial pre-integrated predicted pose for the current frame. The sampling points in the visible area are used as the set of points to be registered, and the projection of the edge point set of the three-dimensional model of the original Chinese prototype exhibit in the semantic map of the display case under the inertial pre-integration predicted pose is used as the target point set. Perform iterative nearest-point algorithm for local registration to obtain the pose to be verified in the current frame; Calculate the reprojection error of the prototype exhibit of the pose to be verified, and verify the pose to be verified based on the reprojection error; When the verification is successful, the pose to be verified is taken as the current real-time pose and tracking is resumed, and the real-time pose of the user device is updated. If the verification fails, extract all visible display case border lines and exhibit silhouette features in the current frame, perform global feature matching with the display case semantic map, and re-solve the initial pose.

[0048] In specific implementation, the inertial measurement unit (IMU) raw data during the tracking interruption period is pre-integrated to generate the inertial pre-integration predicted pose for the current frame. This can be achieved in the following way: obtain all IMU raw data between the image acquisition time corresponding to the previous valid pose and the image acquisition time of the current frame; perform numerical integration on the acceleration and angular velocity data respectively to calculate the rotation increment and translation increment of the device during this period; multiply the rotation increment and translation increment with the previous valid pose to obtain the inertial pre-integration predicted pose for the current frame; during the pre-integration process, the median integration method is used to improve the integration accuracy, and the zero bias of the IMU is estimated and compensated online.

[0049] In specific implementation, the sampling points in the visible area are used as the set of points to be registered, and the projection of the edge point set of the cultural and creative prototype exhibit 3D model in the semantic map of the display case under the inertial pre-integration predicted pose is used as the target point set. This can be achieved in the following way: for example, read the coordinates of all edge vertices of the cultural and creative prototype exhibit 3D model from the semantic map of the display case to form the edge point set of the 3D model; use the inertial pre-integration predicted pose and camera intrinsic parameter matrix to project the edge point set of the 3D model onto the current frame image plane to obtain the two-dimensional projection point set, which is the target point set; use the sampling points in the visible area as the set of points to be registered, and the number of points in the set of points to be registered is consistent with the number of points in the target point set.

[0050] In specific implementation, the iterative nearest point algorithm is used for local registration. The pose to be verified in the current frame can be obtained in the following way: In each iteration, for each sampling point in the set of points to be registered, the point with the closest Euclidean distance to it is searched in the target point set to establish a point pair correspondence; the rotation matrix and translation vector that minimize the sum of squared distances between all point pairs are calculated; the coordinates of the set of points to be registered are updated using the calculated rotation matrix and translation vector; the above iterative process is repeated until the total distance change is less than the preset convergence threshold or the number of iterations reaches the preset upper limit; the rotation matrix and translation vector obtained at the end of the iteration are multiplied with the inertial pre-integration predicted pose to obtain the pose to be verified in the current frame.

[0051] In specific implementation, the reprojection error of the cultural and creative prototype exhibit in the pose to be verified is calculated. Verification of the pose based on the reprojection error can be achieved in the following way: For example, all vertices of the 3D model of the cultural and creative prototype exhibit stored in the semantic map of the exhibition hall are projected onto the current frame image plane through the pose to be verified and the camera intrinsic parameter matrix, obtaining the predicted pixel coordinates of each vertex in the current frame; the distance between the predicted pixel coordinates of each vertex and the pixel coordinates of the nearest point on the outer boundary contour of the silhouette feature of the cultural and creative prototype exhibit is calculated, and the average of the corresponding distances of all vertices is taken as the reprojection error of the cultural and creative prototype exhibit in the pose to be verified; the reprojection error is compared with a preset verification threshold. If the reprojection error is less than the preset verification threshold, then... The verification is considered successful if the reprojection error is greater than or equal to a preset verification threshold, and unsuccessful if the reprojection error is greater than or equal to a preset verification threshold. The preset verification threshold is set based on the vertex density of the 3D model of the cultural and creative prototype exhibit in the semantic map of the showcase and the noise level of the current frame image. The higher the vertex density and the lower the image noise, the smaller the preset verification threshold can be set to perform more stringent pose verification. Conversely, it can be appropriately increased to tolerate a certain degree of geometric deviation. This application does not limit the specific value of the threshold. In other embodiments, the intersection-union ratio (IUU) of the projected outline of the cultural and creative prototype exhibit in the pose to be verified and the outer boundary outline of the silhouette feature of the cultural and creative prototype exhibit can also be calculated, and whether the IUU is greater than a preset IUU threshold can be used as the verification basis. This application does not limit this method.

[0052] In specific implementation, upon successful verification, the pose to be verified is used as the current real-time pose and tracking resumes. Updating the user device's real-time pose can be achieved in the following ways: for example, assigning the rotation matrix and translation vector contained in the pose to be verified to the real-time pose variable of the current frame, overwriting the real-time pose of the previous frame that was stalled due to tracking interruption; simultaneously, re-enabling the inter-frame pixel displacement tracking process described in step three, using the extracted display case border features and cultural and creative prototype silhouette features of the current frame as the starting point for tracking the next frame. In this embodiment, when the verification fails, all visible display case border segments and exhibit silhouette features in the current frame are extracted and globally matched with the display case semantic map to resolve the initial pose. Specifically, this involves: extracting all visible display case border segments and cultural and creative prototype exhibit silhouette features in the current frame; globally matching the extracted features with the display case 3D border information and exhibit 3D model information stored in the display case semantic map; resolving the initial pose using the matched 2D-3D correspondence; and if the global matching fails, displaying a prompt message on the screen to guide the user to align the camera with the complete border area of ​​the display case.

[0053] In practice, before executing the aforementioned local registration and restoration process, a precondition judgment must be performed first, namely: calculating the visible area ratio of the silhouette features of the current frame's cultural and creative prototype exhibit (the ratio of the visible outline arc length to the complete outline arc length of the exhibit stored in the semantic map of the display case), and obtaining the duration of the tracking interruption; that is, only when the visible area ratio is greater than 50% and the tracking interruption time is less than 1 second, will the subsequent local registration and pose restoration operations be executed; if any condition is not met, the global relocalization process is directly triggered, skipping the local registration step, to prevent misidentification of the local outline and registration to the wrong position, while ensuring that the initial pose predicted by the inertial sensor is still within the range that the algorithm can correct, so that the registration can be successful.

[0054] It should be noted that the real-time pose described in this application refers to the spatial positioning parameters, which are continuously updated and output by the inter-frame tracking or occlusion recovery process during system operation, and represent the rotation matrix and translation vector of the user equipment camera coordinate system relative to the display case world coordinate system at the current moment.

[0055] In step 105, based on the real-time pose, the virtual model of the cultural and creative product is rendered onto the spatial position corresponding to the silhouette feature of the cultural and creative prototype exhibit within the display case, thereby generating a display screen.

[0056] In some embodiments, rendering the virtual model of the cultural and creative product onto the spatial position corresponding to the silhouette feature of the prototype cultural and creative exhibit within the display case, based on the real-time pose, to generate the display screen can be achieved through the following steps: Based on the real-time pose, determine the spatial position of the 3D model of the cultural and creative prototype exhibit corresponding to the silhouette feature of the cultural and creative prototype exhibit in the semantic map of the display case; Based on the spatial position of the three-dimensional model of the cultural and creative prototype exhibit, the placement position of the virtual model of the cultural and creative product is determined, and the model transformation matrix of the virtual model of the cultural and creative product is constructed. A projection matrix is ​​constructed using the real-time pose and camera intrinsic parameter matrix to project the virtual model of the cultural and creative product from the three-dimensional world coordinate system to the two-dimensional image plane, generating a rendering layer. The rendered layer is overlaid and combined with the real-time image of the current frame to generate the display screen.

[0057] In specific implementation, the spatial position of the 3D model of the cultural and creative prototype exhibit corresponding to the silhouette feature of the cultural and creative prototype exhibit in the semantic map of the display case, based on the real-time pose, can be determined in the following way: for example, the pre-stored 3D model of the cultural and creative prototype exhibit and its bounding box coordinates in the local coordinate system of the display case are read from the semantic map of the display case; using the rotation matrix and translation vector contained in the real-time pose, the bounding box coordinates of the 3D model of the cultural and creative prototype exhibit in the local coordinate system of the display case are transformed to the world coordinate system to obtain the spatial position of the 3D model of the cultural and creative prototype exhibit in the world coordinate system. This spatial position is the area where the silhouette feature of the cultural and creative prototype exhibit is located in the real 3D space; if the semantic map of the display case contains multiple 3D models of cultural and creative prototype exhibits, then based on the pixel region position of the silhouette feature of the cultural and creative prototype exhibit in the current frame image, combined with the exhibit relative topological position information stored in the semantic map of the display case, the 3D model of the cultural and creative prototype exhibit with the highest reprojection overlap with the pixel region and conforming to the topological position relationship is selected as the matching object, and its spatial position is taken as the output. This application does not limit this.

[0058] In specific implementation, the placement position of the virtual model of the cultural and creative product is determined based on the spatial position of the 3D model of the prototype exhibit, and the model transformation matrix of the virtual model of the cultural and creative product is constructed. This can be achieved in the following way: taking the spatial position of the 3D model of the prototype exhibit as a reference, offsetting a preset distance along a preset direction outside the bounding box of the 3D model of the prototype exhibit by a preset distance, and using the offset coordinates as the placement position of the virtual model of the cultural and creative product. The preset direction can be selected as the right side, top, or front of the 3D model of the prototype exhibit. The preset distance is set based on the actual size of the display case and the aesthetic display requirements of the virtual model of the cultural and creative product. The larger the display case space, the better. The more emphasis is placed on visual separation, the larger the preset distance can be set; conversely, the distance should be appropriately reduced to prevent the virtual model from exceeding the boundaries of the display case. After determining the placement position, a model transformation matrix is ​​constructed from the model coordinate system of the cultural and creative product virtual model to the world coordinate system. This model transformation matrix includes translation transformation to move the cultural and creative product virtual model from the origin of the model coordinate system to the placement position, rotation transformation to adjust the orientation of the cultural and creative product virtual model to face the user device, and scaling transformation to coordinate with the size of the cultural and creative prototype exhibit in the display case. This completes the positioning and posture setting of the cultural and creative product virtual model in the world coordinate system.

[0059] It should be noted that the model transformation matrix mentioned in this application refers to the mathematical transformation expression used to transform the virtual model of cultural and creative products from its own model coordinate system to the world coordinate system. It is used to translate, rotate and scale the virtual model according to the calculated placement position and posture, so that it appears in the target display position in the display case with the correct spatial position and size.

[0060] In specific implementation, a projection matrix is ​​constructed using the real-time pose and camera intrinsic matrix to project the virtual model of the cultural and creative product from the three-dimensional world coordinate system to the two-dimensional image plane. The generation of the rendering layer can be achieved in the following ways: For example, the rotation matrix and translation vector contained in the real-time pose are combined into a camera extrinsic matrix, and the camera extrinsic matrix is ​​multiplied by the camera intrinsic matrix to obtain the complete projection matrix; the model transformation matrix is ​​applied to the coordinates of each vertex of the virtual model of the cultural and creative product to obtain the position of each vertex in the world coordinate system; the positions of each vertex in the world coordinate system are projected onto the two-dimensional image plane through the projection matrix to obtain the pixel coordinates of each vertex on the image plane; according to the pixel coordinates of each vertex and the preset triangular facet topological connection relationship, the surface of the virtual model of the cultural and creative product is rasterized and textured to generate a rendering layer containing the appearance of the virtual model of the cultural and creative product; wherein, during the rasterization process, a depth buffer is used for visibility determination to ensure that the foreground and background occlusion relationship can be correctly presented when the virtual model of the cultural and creative product is partially occluded by transparent or semi-transparent foreground objects such as display case glass.

[0061] It should be noted that the rendering layer mentioned in this application refers to a two-dimensional image layer with the same size as the current frame image, generated by rasterization and texture mapping of the virtual model of the cultural and creative product after model transformation and projection transformation.

[0062] In specific implementation, the rendering layer is superimposed and synthesized with the real-time image of the current frame to generate the display screen. This can be achieved in the following ways: For example, the real-time image of the current frame after distortion correction and histogram equalization preprocessing is obtained as the bottom background; the rendering layer is superimposed on the bottom background in a pixel coordinate aligned manner. For each pixel in the rendering layer, if the pixel belongs to the visible area of ​​the cultural and creative product virtual model, the pixel value of the corresponding position of the bottom background is replaced with the color value of the rendering layer pixel; if the pixel belongs to a non-model area, the original pixel value of the bottom background is retained, thereby generating a composite image that simultaneously contains the real display case scene and the cultural and creative product virtual model as the display screen; during the superimposition and synthesis process, the rendering layer can be cropped in combination with the foreground mask of the silhouette feature of the cultural and creative prototype exhibit, and only the part of the cultural and creative product virtual model that falls outside the specified area of ​​the foreground mask is superimposed and rendered, so that the cultural and creative product virtual model visually appears to be placed next to the cultural and creative prototype exhibit in the display case rather than obscuring the real cultural and creative prototype exhibit. This application does not limit the specific implementation of this superimposition and synthesis method.

[0063] It should be noted that the overlay synthesis described in this application refers to the image processing process of aligning the rendering layer containing the virtual model of the cultural and creative product with the real real-time image of the current frame according to pixel coordinates and merging them pixel by pixel, so as to seamlessly embed the virtual three-dimensional cultural and creative product into the real museum display scene.

[0064] Furthermore, in another aspect of this application, in some embodiments, this application provides an augmented reality-based design and display system for cultural and creative products, with reference to... Figure 4 The figure is a schematic diagram of the structure of an augmented reality-based cultural and creative product design and display system according to some embodiments of this application. The augmented reality-based cultural and creative product design and display system 200 includes: a data acquisition module 201, a processing module 202, and an execution module 203, which are described below: The acquisition module 201 in this application is mainly used to acquire real-time images of the cultural and creative display case and raw data from the inertial measurement unit of the user equipment, and to extract the display case border line features and the silhouette features of the cultural and creative prototype exhibits from the real-time images. Processing module 202, in this application, is mainly used to match the features of the display case border line and the silhouette features of the cultural and creative prototype exhibit with the pre-constructed semantic map of the display case to determine the initial pose of the user device relative to the display case; In addition, the processing module 202 in this application is also used to continuously track the inter-frame pixel displacement of the display case border line features and the silhouette features of the cultural and creative prototype exhibits during the process of the user moving the device, and update the real-time pose of the user device with the initial pose as the starting point by combining the pre-integrated data of the inertial measurement unit. In addition, the processing module 202 in this application is also used to extract the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit when the number of effective border line features is lower than a preset threshold, causing the tracking to be interrupted. The visible area is then locally registered with the projection of the edge point set of the three-dimensional model of the cultural and creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is restored and verified using the inertial pre-integration predicted pose as the initial value. If the verification is successful, the tracking is resumed and the real-time pose is updated. Otherwise, the global relocalization process is triggered. The execution module 203 in this application is mainly used to render the virtual model of the cultural and creative product onto the spatial position corresponding to the silhouette feature of the cultural and creative prototype exhibit in the display case according to the real-time pose, and generate a display screen.

[0065] In addition, this application also provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to acquire the code and execute the above-described augmented reality-based cultural and creative product design and display method.

[0066] In some embodiments, reference Figure 5 This figure is an internal structural diagram of a computer device implementing an augmented reality-based cultural and creative product design and display method according to some embodiments of this application. The augmented reality-based cultural and creative product design and display method in the above embodiments can be implemented through... Figure 5 The computer device shown is used to implement this, and the computer device 300 includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0067] The processor 301 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the augmented reality-based cultural and creative product design and display method in this application.

[0068] The communication bus 302 is used to transmit information between the aforementioned components.

[0069] Memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CDROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 303 may exist independently and be connected to processor 301 via communication bus 302. Memory 303 may also be integrated with processor 301.

[0070] The memory 303 stores program code for executing the solution of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the augmented reality-based cultural and creative product design and display method can be implemented by the processor 301 and one or more software modules in the program code in the memory 303.

[0071] Communication interface 304 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0072] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core processor or a multi-core processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0073] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device may be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0074] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described augmented reality-based cultural and creative product design and display method.

[0075] In summary, the augmented reality-based cultural and creative product design and display method and system disclosed in this application collects real-time images of the cultural and creative display case and raw data from the inertial measurement unit of the user device. It then extracts the display case border line features and the silhouette features of the cultural and creative prototype exhibit from the real-time images. These features are matched with a pre-constructed semantic map of the display case to determine the initial pose of the user device relative to the display case. During the user's mobile device operation, the inter-frame pixel displacement of the display case border line features and the silhouette features of the cultural and creative prototype exhibit is continuously tracked. Combined with pre-integrated data from the inertial measurement unit, the real-time pose of the user device is updated starting from the initial pose. When the number of effective borderline features falls below a preset threshold, causing tracking to be interrupted, the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit is extracted. The visible area is then locally registered with the projection of the edge point set of the 3D model of the cultural and creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is recovered and verified using the inertial pre-integration predicted pose as the initial value. If the verification is successful, tracking is resumed and the real-time pose is updated; otherwise, a global relocalization process is triggered. Based on the real-time pose, the virtual model of the cultural and creative product is rendered to the spatial position corresponding to the silhouette feature of the cultural and creative prototype exhibit within the display case, generating a display screen. Pose correction can be performed through the visible local area of ​​the silhouette of the cultural and creative prototype exhibit to maintain display stability.

[0076] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0077] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for designing and displaying cultural and creative products based on augmented reality, characterized in that, Includes the following steps: Real-time images of the cultural and creative product display cases and raw data from the inertial measurement unit of the user equipment are collected, and the features of the display case border lines and the silhouette features of the cultural and creative product prototypes in the real-time images are extracted. The display case border features and the silhouette features of the cultural and creative prototype exhibits are matched with the pre-constructed semantic map of the display case to determine the initial pose of the user device relative to the display case. During the process of the user moving the device, the inter-frame pixel displacement of the display case border line features and the silhouette features of the cultural and creative prototype exhibit is continuously tracked. Combined with the pre-integrated data of the inertial measurement unit, the real-time pose of the user device is updated from the initial pose. When the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit is extracted. The visible area is locally registered with the projection of the edge point set of the 3D model of the cultural and creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is restored and verified with the inertial pre-integration predicted pose as the initial value. If the verification is successful, tracking is resumed and the real-time pose is updated. Otherwise, the global relocalization process is triggered. Based on the real-time pose, the virtual model of the cultural and creative product is rendered onto the spatial position corresponding to the silhouette feature of the prototype cultural and creative exhibit within the display case, generating a display screen.

2. The method as described in claim 1, characterized in that, Matching the display case border features and the silhouette features of the cultural and creative prototype exhibit with a pre-constructed semantic map of the display case to determine the initial pose of the user device relative to the display case specifically includes: The features of the display case border line are matched with the 3D border information of the display case stored in the pre-constructed semantic map of the display case to establish the correspondence between the 2D line segments in the image and the 3D border in the world coordinate system. The initial pose of the user equipment relative to the display case is calculated based on the aforementioned correspondence. The silhouette features of the cultural and creative prototype exhibit are matched with the projection contour of the 3D model of the cultural and creative prototype exhibit in the semantic map of the display case under the initial pose, and the initial pose is corrected according to the matching result to determine the initial pose of the user device relative to the display case.

3. The method as described in claim 1, characterized in that, During the process of the user moving the device, the inter-frame pixel displacement of the display case border features and the silhouette features of the cultural and creative prototype exhibit is continuously tracked. Combined with the pre-integrated data of the inertial measurement unit, the real-time pose of the user device is updated starting from the initial pose. Specifically, this includes: During the process of the user's mobile device, a set of feature points for tracking is extracted from the display case border line features and the silhouette features of the cultural and creative prototype exhibits in the current frame; Based on the feature point set of the previous frame, the pixel coordinates of each feature point in the feature point set are tracked in the current frame to obtain the inter-frame pixel displacement of the feature point set. The raw data from the inertial measurement unit is pre-integrated to generate a predicted value of the relative pose increment between adjacent frames. Based on the inter-frame pixel displacement and the real-time pose of the previous frame, the corresponding 3D points in the semantic map of the display case are projected onto the current frame to establish the correspondence between the 3D points and the 2D tracking points. Using the inertial pre-integration relative pose increment prediction value as the initial value, the real-time pose of the current frame is calculated using the correspondence between the three-dimensional points and the two-dimensional tracking points, and the real-time pose of the user equipment is updated.

4. The method as described in claim 1, characterized in that, When the number of effective border line features falls below a preset threshold, causing tracking to be interrupted, the visible area of ​​the silhouette features of the cultural and creative prototype exhibit is extracted specifically including: When the number of effective border line features is lower than a preset threshold, causing tracking to be interrupted, the foreground mask is obtained from the silhouette features of the cultural and creative prototype exhibit in the current frame to determine the pixel area occupied by the cultural and creative prototype exhibit in the image. Extract the outer boundary contour of the silhouette features of the cultural and creative prototype exhibit within the pixel area to obtain the set of visible contour points; Uniform sampling is performed on the set of visible contour points to extract the visible area of ​​the silhouette features of the cultural and creative prototype exhibit.

5. The method as described in claim 1, characterized in that, The visible area is locally registered with the projection of the edge point set of the 3D model of the creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is recovered and verified using the inertial pre-integration predicted pose as the initial value. If the verification is successful, tracking is resumed and the real-time pose is updated. Otherwise, a global relocalization process is triggered as the real-time pose, which specifically includes: The raw data from the inertial measurement unit during the tracking interruption period is pre-integrated to generate the inertial pre-integrated predicted pose for the current frame. The sampling points in the visible area are used as the set of points to be registered, and the projection of the edge point set of the three-dimensional model of the original Chinese prototype exhibit in the semantic map of the display case under the inertial pre-integration predicted pose is used as the target point set. Perform iterative nearest-point algorithm for local registration to obtain the pose to be verified in the current frame; Calculate the reprojection error of the prototype exhibit of the pose to be verified, and verify the pose to be verified based on the reprojection error; When the verification is successful, the pose to be verified is taken as the current real-time pose and tracking is resumed, and the real-time pose of the user device is updated. If the verification fails, extract all visible display case border lines and exhibit silhouette features in the current frame, perform global feature matching with the display case semantic map, and re-solve the initial pose.

6. The method as described in claim 1, characterized in that, Based on the real-time pose, the virtual model of the cultural and creative product is rendered onto the spatial position corresponding to the silhouette feature of the prototype cultural and creative exhibit within the display case, generating a display screen, specifically including: Based on the real-time pose, determine the spatial position of the 3D model of the cultural and creative prototype exhibit corresponding to the silhouette feature of the cultural and creative prototype exhibit in the semantic map of the display case; Based on the spatial position of the three-dimensional model of the cultural and creative prototype exhibit, the placement position of the virtual model of the cultural and creative product is determined, and the model transformation matrix of the virtual model of the cultural and creative product is constructed. A projection matrix is ​​constructed using the real-time pose and camera intrinsic parameter matrix to project the virtual model of the cultural and creative product from the three-dimensional world coordinate system to the two-dimensional image plane, generating a rendering layer. The rendered layer is overlaid and combined with the real-time image of the current frame to generate the display screen.

7. The method as described in claim 1, characterized in that, The semantic map of the display case includes three-dimensional spatial information of the display case and the prototype cultural and creative exhibits inside, as well as the relative topological position information of the exhibits.

8. An augmented reality-based design and display system for cultural and creative products, comprising: The acquisition module is used to acquire real-time images of the cultural and creative display case and raw data from the inertial measurement unit of the user equipment, and to extract the display case border features and the silhouette features of the cultural and creative prototype exhibits from the real-time images. The processing module is used to match the border features of the display case and the silhouette features of the cultural and creative prototype exhibit with the pre-built semantic map of the display case to determine the initial pose of the user device relative to the display case. The processing module is also used to continuously track the inter-frame pixel displacement of the display case border line features and the silhouette features of the cultural and creative prototype exhibits during the user's mobile device operation, and update the real-time pose of the user device with the initial pose as the starting point by combining the pre-integrated data of the inertial measurement unit. The processing module is also used to extract the visible area of ​​the silhouette feature of the cultural and creative prototype exhibit when the number of effective border line features is lower than a preset threshold, causing the tracking to be interrupted. The visible area is then locally registered with the projection of the edge point set of the three-dimensional model of the cultural and creative prototype exhibit in the semantic map of the display case under the previous effective pose. The current pose is restored and verified using the inertial pre-integration predicted pose as the initial value. If the verification is successful, the tracking is resumed and the real-time pose is updated. Otherwise, the global relocalization process is triggered. The execution module is used to render the virtual model of the cultural and creative product onto the spatial position corresponding to the silhouette feature of the cultural and creative prototype exhibit within the display case, based on the real-time pose, and generate a display screen.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the augmented reality-based cultural and creative product design and display method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the augmented reality-based cultural and creative product design and display method as described in any one of claims 1 to 7.

Images