AR scene real-time alignment method of lightweight CNN

Abstract

The invention discloses an AR scene real-time alignment method of a lightweight CNN, and belongs to the field of augmented reality image processing. The method comprises the following steps: firstly, acquiring an AR scene multi-view real image, virtual object three-dimensional model data and IMU inertial measurement data; a lightweight CNN containing an enhanced Ghost bottleneck structure is constructed, low-level structures and high-level semantic features of the image are extracted in a layered mode, and scene key feature points and a reliability graph are generated through feature pyramid fusion; fusing multiple types of data by taking the feature points as a core, and generating an initial alignment transformation matrix through geometric correction; and finally, adjusting the posture of the virtual object, and smoothly optimizing and correcting the matrix through a time sequence to finish real-time alignment. The method balances model lightweight and feature extraction capability, reduces deviation caused by motion, improves alignment accuracy and stability, ensures natural fusion of virtual and real scenes, and adapts to application requirements of mobile AR equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality image processing, in particular to a real-time alignment method of AR scene based on lightweight CNN. BACKGROUND

[0002] With the wide application of augmented reality (AR) technology in various scenarios, real-time and accurate alignment of real scenes and virtual objects has become one of the core links to support the landing of AR technology. At present, the image alignment technology of AR scene mainly focuses on neural networks, sensor data fusion, feature extraction and matching, etc. The industry generally pursues real-time, accuracy and device adaptability of the alignment method. In the field of technology application, convolutional neural network (CNN) has become the mainstream technology choice for feature processing in AR image alignment due to its strong feature extraction capability. At the same time, in order to adapt to the hardware resource constraints of mobile AR devices, the research and application of lightweight neural network model has become an important trend. In addition, the fusion strategy of inertial measurement unit (IMU) data and image data, virtual object three-dimensional model data, feature pyramid fusion, spatial transformation matrix calculation, time sequence optimization, etc. are also widely used in various alignment schemes, forming a technical framework based on feature extraction, core data fusion and supplementary optimization and correction, which promotes the development of AR alignment technology towards high efficiency and accuracy.

[0003] However, there are still many problems to be solved in the existing real-time image alignment technology of AR scene. The neural network model structure adopted by some alignment methods is not reasonable enough, which cannot effectively balance the model lightweight and feature extraction capability, resulting in either large model parameter quantity and high computational complexity, which is difficult to meet the real-time running demand of mobile AR devices, or simplifying the feature extraction process for the purpose of lightweight, which cannot fully capture the low-level structural features and high-level semantic features of the real scene, thereby affecting the extraction accuracy of the key feature points of the scene, leading to large alignment deviation. At the same time, the existing methods lack systematic design in the fusion application of multiple types of data, and cannot effectively coordinate the virtual object three-dimensional model data and IMU inertial measurement data with the scene feature points as the core. The feature matching process is easily affected by factors such as device motion, resulting in deviation, and lacks a targeted geometric correction mechanism. In addition, some methods do not perform effective time sequence smoothing optimization and correction after obtaining the initial alignment transformation matrix, resulting in unstable motion trajectory of the virtual object in the continuous frame, which is difficult to realize long-term accurate alignment of real scene and virtual object. These problems jointly restrict the application experience and popularization and landing of AR technology. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a real-time alignment method of AR scene based on lightweight CNN.

[0005] The objective of this invention is achieved through the following technical solution: A lightweight CNN-based real-time alignment method for AR scenes is provided, which includes the following steps: S1. Acquire multi-view real-world scene images of AR scenes, 3D model data of virtual objects, and IMU inertial measurement data; S2. Construct a lightweight convolutional neural network with an enhanced Ghost bottleneck structure. Extract low-level structural features and high-level semantic features from real scene images through this network in layers. After feature pyramid fusion, generate key feature points and feature reliability maps of the scene. S3. Taking the key feature points of the scene as the core, the system assists in fusing IMU inertial measurement data and virtual object 3D model data, corrects feature matching deviations through geometric correction, and generates an initial alignment transformation matrix. S4. Adjust the virtual object's pose based on the initial alignment transformation matrix, and correct the matrix parameters using a temporal smoothing optimization algorithm to achieve real-time alignment between the real scene and the virtual object.

[0006] Furthermore, step S1 includes: S1.1. Collect multi-view real scene images from different directions and shooting angles around the core interactive area of ​​the AR scene. Keep the shooting device parameters consistent during the collection process, record the precise timestamp of each image collection moment, and perform preliminary noise filtering processing on the collected images. S1.2. Obtain complete 3D model data of the virtual object, including vertex coordinate data, face index data, and texture mapping data. At the same time, obtain the preset posture parameters of the virtual object, including initial position parameters, initial rotation angle parameters, and initial scaling ratio parameters. S1.3. Real-time acquisition of inertial measurement data from the built-in IMU sensor of the AR device. The IMU inertial measurement data includes three-axis acceleration data, three-axis angular velocity data, and three-axis attitude angle data. The acquired inertial measurement data is timestamped to ensure that it corresponds to the acquisition time of the real scene image.

[0007] Furthermore, step S2 includes: S2.1. Construct a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer, and a parallel output branch. Data transmission links are established between each layer through a fully connected approach. The input preprocessing layer is used for image standardization, the multi-level feature extraction layer is used for extracting features of different dimensions in a hierarchical manner, the enhanced Ghost module is used for feature expansion and optimization, the feature fusion layer is used for multi-scale feature integration, and the parallel output branch is used to generate the target results respectively. S2.2. The pixel values ​​of the real scene image are normalized by the input preprocessing layer, and the pixel values ​​are mapped to the preset range. At the same time, the image size is adjusted according to the input requirements of the lightweight convolutional neural network to ensure that the image size matches the network input channel. The processed image is input into the multi-level feature extraction layer, and the edge and texture low-level structure features at different resolutions are obtained through convolution operations with different kernel sizes. S2.3. The low-level structural features output by the multi-level feature extraction layer are concatenated along the channel dimension and then input into the enhanced Ghost module. Through feature transformation and expansion operations within the module, a rich feature representation containing more detailed information is generated. S2.4. By using a feature pyramid structure fusion layer, features at different scales are fused layer by layer. First, the high-level features are upsampled to make them consistent with the resolution of the low-level features. Then, the features are superimposed and fused. The fused features are input into parallel output branches. Through convolution and activation operations within the branches, key feature points of the scene and feature reliability maps representing the credibility of feature matching are generated respectively.

[0008] Furthermore, step S3 includes: S3.1. Read the feature reliability map output by the lightweight convolutional neural network, extract the confidence value of each pixel in the feature reliability map, set a confidence threshold, and filter out the key feature points of the scene with confidence values ​​higher than the threshold to form a set of high confidence feature points. S3.2. Extract surface feature points from the 3D model data of virtual objects, establish a virtual feature point library, use a feature distance matching algorithm to associate and match feature points in the high confidence feature point set with feature points in the virtual feature point library, calculate the similarity between pairs of feature points, and determine the initial feature correspondence based on the similarity ranking results. S3.3. Auxiliary integration of pre-processed IMU inertial measurement data, calculation of the motion trajectory and attitude change of AR device based on IMU data, construction of motion offset model, geometric correction of feature point positions in the initial feature correspondence based on the model, and correction of feature offset caused by device motion; S3.4. Based on the corrected feature correspondence, extract the coordinate information of the corresponding feature points in the real scene image coordinate system and the virtual object three-dimensional coordinate system, calculate the spatial transformation parameters through the coordinate transformation formula, the spatial transformation parameters include translation parameters, rotation parameters and scaling parameters, and integrate these parameters to generate an initial alignment transformation matrix.

[0009] Furthermore, step S4 includes: S4.1. Apply the translation, rotation and scaling parameters in the initial alignment transformation matrix to the preset posture parameters of the virtual object, and update the position, angle and size of the virtual object in three-dimensional space according to the matrix transformation rules, so that the virtual object can initially adapt to the spatial environment of the real scene. S4.2. Extract the edge contour features of the adjusted virtual object and the edge contour features of the corresponding region of the real scene image, calculate the overlap and distance deviation geometric error between the two types of edge contour features, and quantify the alignment degree between the virtual object and the real scene. S4.3. The initial alignment transformation matrix is ​​corrected by using a temporal smoothing optimization algorithm. First, the historical alignment parameters of multiple consecutive frames are introduced as a reference. The difference between the alignment parameters of the current frame and the historical frames is calculated. An optimization objective function is constructed by combining the geometric error. The minimum value of the optimization objective function is obtained by iteratively solving the problem. S4.4. Incorporate spatiotemporal consistency constraints to ensure that the motion trajectory of virtual objects in consecutive frames conforms to the laws of physical motion, avoiding abrupt displacements. Perform a final verification on the corrected virtual object posture, and output the aligned AR image after the verification passes.

[0010] Furthermore, in step S2.3, the enhanced Ghost module first performs channel recombination and feature aggregation on low-level structural features through 1×1 convolution to generate a high-dimensional and dense intrinsic feature map; then, through a combination of depthwise separable convolution and attention gating units, it performs feature expansion and redundancy suppression on the intrinsic feature map to generate a complementary Ghost feature map; after element-wise weighted fusion of the intrinsic feature map and the Ghost feature map, it outputs a rich feature representation adapted to subsequent fusion through batch normalization and activation function processing. This process embeds the feature extraction link of a lightweight convolutional neural network throughout, enhancing the network's ability to capture scene detail features.

[0011] Furthermore, in step S2.4, the feature pyramid fusion layer adopts a bidirectional fusion strategy that combines top-down and bottom-up approaches. First, the high-level semantic features are upsampled and aligned with the low-level structural features, and then fused channel by channel. Next, the fused features are downsampled and adjusted to supplement detailed feature information. During the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of features at each scale, so that the fused features are more in line with the extraction requirements of key feature points in the scene.

[0012] Furthermore, in step S3.3, the IMU inertial measurement data is first calibrated with timestamps and corrected for zero bias, and an initial estimate of the camera motion is obtained through pre-integration. Then, a motion compensation model is constructed based on the initial estimate to compensate the position of key feature points of the scene output by the lightweight convolutional neural network. By calculating the matching error between the compensated feature points and the features of the virtual object's 3D model, the compensation parameters are iteratively corrected, and finally the corrected feature correspondence is obtained, realizing the auxiliary correction role of IMU data for CNN feature matching.

[0013] Furthermore, in step S4.3, the temporal smoothing optimization algorithm first extracts the changing trend of key scene feature points output by the lightweight convolutional neural network in multiple consecutive frames and constructs a temporal constraint model; then, combined with the geometric error of the current frame, iteratively adjusts the parameters of the initial alignment transformation matrix; during the iteration process, the stability of the CNN features is used to determine the convergence condition of the iteration, and the iteration stops when the feature change is less than the preset range, ensuring that the optimization process is always based on the CNN feature output.

[0014] Furthermore, after step S4, the quality of the aligned AR image is verified. The feature consistency detection branch of the lightweight CNN is used to determine the fusion consistency between the virtual object features and the real scene features. When the consistency does not meet the requirements, the feedback is sent to step S2 to adjust the feature extraction parameters of the lightweight convolutional neural network and to re-perform the feature extraction and subsequent alignment process, forming a closed-loop optimization link with CNN as the core.

[0015] The beneficial effects of this invention are: (1) Based on the lightweight CNN with enhanced Ghost bottleneck structure, feature extraction is performed in layers. Multiple types of data are integrated and optimized by time-series smoothing to achieve real-time alignment between real and virtual objects in AR scenes, taking into account both model lightweighting and alignment accuracy. (2) The feature pyramid integrates multi-scale features, combined with geometric correction and IMU data-assisted correction, to enhance the capture of scene detail features, effectively reduce feature matching deviation, and improve the accuracy of alignment benchmark; (3) The spatiotemporal consistency constraint and the closed-loop optimization link work together to ensure smooth motion of virtual objects in continuous frames, avoid sudden displacement, and further improve alignment stability and the naturalness of virtual scene integration. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the steps of a lightweight CNN-based real-time alignment method for AR scenes; Figure 2 The flowchart illustrates the specific implementation steps of a lightweight CNN-based real-time image alignment method for AR scenes, as provided in this embodiment. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 See Figure 1 This paper provides a lightweight CNN-based real-time alignment method for AR scenes, including the following steps: S1. Acquire multi-view real-world scene images of AR scenes, 3D model data of virtual objects, and IMU inertial measurement data; S2. Construct a lightweight convolutional neural network with an enhanced Ghost bottleneck structure. Extract low-level structural features and high-level semantic features from real scene images through this network in layers. After feature pyramid fusion, generate key feature points and feature reliability maps of the scene. S3. Taking the key feature points of the scene as the core, the system assists in fusing IMU inertial measurement data and virtual object 3D model data, corrects feature matching deviations through geometric correction, and generates an initial alignment transformation matrix. S4. Adjust the virtual object's pose based on the initial alignment transformation matrix, and correct the matrix parameters using a temporal smoothing optimization algorithm to achieve real-time alignment between the real scene and the virtual object.

[0019] In some embodiments, step S3 can first fuse IMU inertial measurement data and virtual object 3D model data to generate a preliminary attitude reference for the virtual object, and then perform correlation matching with key feature points of the scene. Through bidirectional correction, feature matching deviation and virtual object attitude deviation are corrected, and finally an initial alignment transformation matrix is ​​generated. The overall process improves the accuracy of the alignment reference through bidirectional data interaction.

[0020] Step S1 includes: S1.1. Collect multi-view real scene images from different directions and shooting angles around the core interactive area of ​​the AR scene. Keep the shooting device parameters consistent during the collection process, record the precise timestamp of each image collection moment, and perform preliminary noise filtering processing on the collected images. S1.2. Obtain complete 3D model data of the virtual object, including vertex coordinate data, face index data, and texture mapping data. At the same time, obtain the preset posture parameters of the virtual object, including initial position parameters, initial rotation angle parameters, and initial scaling ratio parameters. S1.3. Real-time acquisition of inertial measurement data from the built-in IMU sensor of the AR device. The IMU inertial measurement data includes three-axis acceleration data, three-axis angular velocity data, and three-axis attitude angle data. The acquired inertial measurement data is timestamped to ensure that it corresponds to the acquisition time of the real scene image.

[0021] In some embodiments, step S1.1 may employ a dynamic perspective acquisition strategy, adjusting the acquisition angle and orientation according to the real-time changes of the AR scene, and replacing single-frame noise filtering with multi-frame superposition denoising after acquisition; step S1.2 may additionally acquire physical attribute data of virtual objects for physical constraints during subsequent attitude adjustment; step S1.3 may perform sliding window filtering on the IMU inertial measurement data to further eliminate abnormal data and improve data stability.

[0022] Step S2 includes: S2.1. Construct a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer, and a parallel output branch. Data transmission links are established between each layer through a fully connected approach. The input preprocessing layer is used for image standardization, the multi-level feature extraction layer is used for extracting features of different dimensions in a hierarchical manner, the enhanced Ghost module is used for feature expansion and optimization, the feature fusion layer is used for multi-scale feature integration, and the parallel output branch is used to generate the target results respectively. S2.2. The pixel values ​​of the real scene image are normalized by the input preprocessing layer, and the pixel values ​​are mapped to the preset range. At the same time, the image size is adjusted according to the input requirements of the lightweight convolutional neural network to ensure that the image size matches the network input channel. The processed image is input into the multi-level feature extraction layer, and the edge and texture low-level structure features at different resolutions are obtained through convolution operations with different kernel sizes. S2.3. The low-level structural features output by the multi-level feature extraction layer are concatenated along the channel dimension and then input into the enhanced Ghost module. Through feature transformation and expansion operations within the module, a rich feature representation containing more detailed information is generated. S2.4. By using a feature pyramid structure fusion layer, features at different scales are fused layer by layer. First, the high-level features are upsampled to make them consistent with the resolution of the low-level features. Then, the features are superimposed and fused. The fused features are input into parallel output branches. Through convolution and activation operations within the branches, key feature points of the scene and feature reliability maps representing the credibility of feature matching are generated respectively.

[0023] In some embodiments, step S2.1 can adopt a hybrid link of "fully connected + local skip connection", setting skip connections between adjacent sub-layers of the input preprocessing layer and the feature fusion layer and the multi-level feature extraction layer to reduce feature transfer loss; step S2.3 can replace the enhanced Ghost module with a lightweight attention enhancement module, strengthening effective features through the synergistic effect of channel attention and spatial attention; step S2.4 can use attention-weighted fusion instead of feature superposition fusion, allocating dynamic weights according to the importance of features at different scales.

[0024] Step S3 includes: S3.1. Read the feature reliability map output by the lightweight convolutional neural network, extract the confidence value of each pixel in the feature reliability map, set a confidence threshold, and filter out the key feature points of the scene with confidence values ​​higher than the threshold to form a set of high confidence feature points. S3.2. Extract surface feature points from the 3D model data of virtual objects, establish a virtual feature point library, use a feature distance matching algorithm to associate and match feature points in the high confidence feature point set with feature points in the virtual feature point library, calculate the similarity between pairs of feature points, and determine the initial feature correspondence based on the similarity ranking results. S3.3. Auxiliary integration of pre-processed IMU inertial measurement data, calculation of the motion trajectory and attitude change of AR device based on IMU data, construction of motion offset model, geometric correction of feature point positions in the initial feature correspondence based on the model, and correction of feature offset caused by device motion; S3.4. Based on the corrected feature correspondence, extract the coordinate information of the corresponding feature points in the real scene image coordinate system and the virtual object three-dimensional coordinate system, calculate the spatial transformation parameters through the coordinate transformation formula, the spatial transformation parameters include translation parameters, rotation parameters and scaling parameters, and integrate these parameters to generate an initial alignment transformation matrix.

[0025] In some embodiments, step S3.2 can use the feature vector angle matching algorithm instead of the feature distance matching algorithm, and determine the similarity by calculating the cosine value of the angle between feature vectors; step S3.3 can construct an extended Kalman filter model, tightly couple and fuse IMU data with feature point location data, and simultaneously complete motion estimation and geometric correction; step S3.4 can solve the spatial transformation parameters by the least squares method, and combine robust estimation to eliminate the influence of abnormal corresponding points on parameter calculation.

[0026] Step S4 includes: S4.1. Apply the translation, rotation and scaling parameters in the initial alignment transformation matrix to the preset posture parameters of the virtual object, and update the position, angle and size of the virtual object in three-dimensional space according to the matrix transformation rules, so that the virtual object can initially adapt to the spatial environment of the real scene. S4.2. Extract the edge contour features of the adjusted virtual object and the edge contour features of the corresponding region of the real scene image, calculate the overlap and distance deviation geometric error between the two types of edge contour features, and quantify the alignment degree between the virtual object and the real scene. S4.3. The initial alignment transformation matrix is ​​corrected by using a temporal smoothing optimization algorithm. First, the historical alignment parameters of multiple consecutive frames are introduced as a reference. The difference between the alignment parameters of the current frame and the historical frames is calculated. An optimization objective function is constructed by combining the geometric error. The minimum value of the optimization objective function is obtained by iteratively solving the problem. S4.4. Incorporate spatiotemporal consistency constraints to ensure that the motion trajectory of virtual objects in consecutive frames conforms to the laws of physical motion, avoiding abrupt displacements. Perform a final verification on the corrected virtual object posture, and output the aligned AR image after the verification passes.

[0027] In some embodiments, step S4.2 may include photometric error calculation, which supplements the quantization alignment degree by comparing the pixel brightness distribution differences between the virtual object and the corresponding area of ​​the real scene; step S4.3 may use gradient descent algorithm to replace temporal smoothing optimization algorithm, and directly iteratively correct matrix parameters with the goal of minimizing geometric error; step S4.4 may incorporate photometric consistency constraints to ensure that the lighting effects of the virtual object and the real scene match, thereby improving the naturalness of the fusion.

[0028] In step S2.3, the enhanced Ghost module first performs channel recombination and feature aggregation on low-level structural features through 1×1 convolution to generate a high-dimensional and dense intrinsic feature map; then, it performs feature expansion and redundancy suppression on the intrinsic feature map by combining depthwise separable convolution with attention gating units to generate a complementary Ghost feature map; after element-wise weighted fusion of the intrinsic feature map and the Ghost feature map, it outputs a rich feature representation adapted to subsequent fusion through batch normalization and activation function processing. This process embeds the feature extraction link of a lightweight convolutional neural network throughout, enhancing the network's ability to capture scene detail features.

[0029] In some embodiments, the enhanced Ghost module can first perform channel splitting and local aggregation on low-level structural features through grouped convolution to generate multiple sets of low-dimensional intrinsic feature maps; then, by combining point convolution with spatial attention units, it can perform targeted expansion and redundancy suppression on each set of intrinsic feature maps to generate multiple sets of Ghost feature maps; after channel concatenation of each set of intrinsic feature maps and the corresponding Ghost feature maps, it can filter key feature channels through a global attention module, and then process them through batch normalization and activation functions to output rich feature representations, further improving the relevance and effectiveness of the features.

[0030] In step S2.4, the feature pyramid fusion layer adopts a bidirectional fusion strategy that combines top-down and bottom-up approaches. First, the high-level semantic features are upsampled and aligned with the low-level structural features, and then fused channel by channel. Next, the fused features are downsampled and adjusted to supplement detailed feature information. During the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of features at each scale, so that the fused features are more in line with the extraction requirements of key feature points in the scene.

[0031] In some embodiments, the feature pyramid fusion layer can adopt a reverse fusion order of bottom-up and top-down. First, the low-level structural features are aligned with the high-level semantic features through downsampling and then fused element-wise to highlight key features. Then, the fused features are upsampled and adjusted to supplement global feature information. During the fusion process, the weights of the features in each region are dynamically allocated through the spatial attention branch to focus on the feature fusion of key regions of the scene, making the generated key feature points of the scene more accurately located.

[0032] In step S3.3, the IMU inertial measurement data is first calibrated with timestamps and corrected for zero bias. An initial estimate of the camera motion is obtained through pre-integration. Then, a motion compensation model is constructed based on the initial estimate to compensate the position of key feature points of the scene output by the lightweight convolutional neural network. By calculating the matching error between the compensated feature points and the features of the 3D model of the virtual object, the compensation parameters are iteratively corrected to finally obtain the corrected feature correspondence, thereby realizing the auxiliary correction role of IMU data for CNN feature matching.

[0033] In some embodiments, the timestamps of the IMU inertial measurement data and the real scene image can be calibrated at the sub-millisecond level first, and a continuous estimate of the camera motion can be obtained by pre-integration through a sliding window. Then, an adaptive motion compensation model can be constructed based on the continuous estimate, and the compensation coefficients can be dynamically adjusted according to the spatial position of the feature points to perform regional position compensation for the key feature points of the scene. By calculating the Euclidean distance error and cosine similarity error between the compensated feature points and the features of the virtual object's 3D model, the compensation parameters can be corrected by a weighted iterative method to further improve the accuracy of the feature correspondence.

[0034] In step S4.3, the temporal smoothing optimization algorithm first extracts the changing trend of key scene feature points output by the lightweight convolutional neural network in multiple consecutive frames and constructs a temporal constraint model; then, combined with the geometric error of the current frame, iteratively adjusts the parameters of the initial alignment transformation matrix; during the iteration process, the stability of the CNN features is used to determine the convergence condition of the iteration, and the iteration stops when the feature change is less than the preset range, ensuring that the optimization process is always based on the CNN feature output.

[0035] In some embodiments, the temporal smoothing optimization algorithm can first extract the trend of alignment parameter changes in multiple consecutive frames within a sliding window, construct a temporal constraint model based on time weights, with the alignment parameter weights of recent frames being higher than those of distant frames; then, by combining the geometric error and photometric error of the current frame, a multi-objective optimization objective function is constructed; when iteratively adjusting the parameters of the initial alignment transformation matrix, the convergence condition is determined by the change in the objective function, and the iteration stops when the difference between the objective functions of two consecutive iterations is less than a preset range, thus balancing optimization efficiency and alignment accuracy.

[0036] After step S4, the quality of the aligned AR image is verified. The feature consistency detection branch of the lightweight CNN is used to determine the fusion consistency between the virtual object features and the real scene features. When the consistency does not meet the requirements, the feedback is sent to step S2 to adjust the feature extraction parameters of the lightweight convolutional neural network and to re-perform the feature extraction and subsequent alignment process, forming a closed-loop optimization link with CNN as the core.

[0037] In some embodiments, when verifying the quality of aligned AR images, the feature consistency detection branch and pixel consistency detection branch of a lightweight CNN can work together. The feature consistency detection branch judges the fusion of high-level semantic features, and the pixel consistency detection branch judges the matching of low-level pixel distribution. When either branch detects that the consistency does not meet the requirements, it feeds back to steps S2 and S3, and at the same time adjusts the feature extraction parameters of the lightweight convolutional neural network and the parameters of the feature matching algorithm, and re-performs the feature extraction, matching and subsequent alignment processes to form a more comprehensive closed-loop optimization link and further improve the alignment stability.

[0038] Example 2 This embodiment provides a specific implementation process for a lightweight CNN-based real-time image alignment method for AR scenes. This embodiment achieves accurate real-time alignment between real scenes and virtual objects through systematic data acquisition, feature extraction, data fusion, and optimization adjustments. Figure 2 As shown, the specific steps and implementation process are as follows: S1. Acquire multi-view real-world scene images, 3D model data of virtual objects, and IMU inertial measurement data of AR scenes: S1.1. Collect multi-view real-world scene images from different orientations and shooting angles around the core interactive area of ​​the AR scene: During the acquisition process, the optical parameters and imaging mode of the shooting equipment should be kept consistent to avoid differences in image features that are not due to the scene itself caused by changes in equipment parameters; the precise timestamp of each image acquisition moment should be recorded, and the timestamp should be accurate to the instant of image acquisition to ensure time synchronization during subsequent data fusion; at the same time, the acquired images should be subjected to preliminary noise filtering processing, and random noise caused by environmental interference and equipment shaking during the acquisition process should be removed through basic image smoothing operations, so as to preserve the original structure and feature information of the scene in the image; S1.2. Obtain the complete 3D model data and preset posture parameters of the virtual object: The acquired 3D model data of the virtual object includes vertex coordinate data, face index data, and texture mapping data. Vertex coordinate data is used to characterize the 3D spatial structure of the virtual object, face index data is used to define the connection relationship between vertices to form the surface morphology of the virtual object, and texture mapping data is used to present the surface texture details of the virtual object. At the same time, the preset posture parameters of the virtual object are acquired. The preset posture parameters include initial position parameters, initial rotation angle parameters, and initial scaling ratio parameters. The initial position parameters are used to set the initial placement position of the virtual object in the AR scene, the initial rotation angle parameters are used to define the spatial orientation of the virtual object in its initial state, and the initial scaling ratio parameters are used to match the size ratio relationship between the virtual object and the real scene. S1.3. Real-time acquisition and preprocessing of inertial measurement data from the AR device's built-in IMU sensor: Inertial measurement data output from the built-in IMU sensor of the AR device is acquired in real time. The IMU inertial measurement data includes three-axis acceleration data, three-axis angular velocity data, and three-axis attitude angle data. The three-axis acceleration data reflects the linear motion state of the AR device, the three-axis angular velocity data represents the rotational motion state of the AR device, and the three-axis attitude angle data describes the attitude information of the AR device in space. The acquired inertial measurement data is timestamped to ensure that the timestamps of the IMU data correspond one-to-one with the acquisition time of the real scene image, ensuring the synchronization of the two types of data in the time dimension. At the same time, the IMU data is subjected to preliminary outlier removal processing to remove abnormal data points caused by instantaneous sensor failure or severe external interference, ensuring the reliability of the inertial measurement data. S2. Construct a lightweight convolutional neural network with an enhanced Ghost bottleneck structure. This network extracts low-level structural features and high-level semantic features from real-world scene images in layers. After fusion via feature pyramids, it generates key scene feature points and feature reliability maps. S2.1. Construct a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer, and a parallel output branch, and complete the training: When building a lightweight convolutional neural network, data transmission links are established between layers through full connectivity to ensure the effective transfer of feature data between modules. The input preprocessing layer is specifically used for image standardization, providing data input with uniform specifications for subsequent feature extraction. The multi-level feature extraction layer extracts image features of different dimensions in a hierarchical manner, progressing from basic structural features to complex semantic features. The enhanced Ghost module is used for feature expansion and optimization, improving feature expressiveness while controlling the number of parameters. The feature fusion layer integrates features at multiple scales, aggregating effective features at different resolutions. The parallel output branch generates two types of target results: key scene feature points and feature reliability maps. The network's hierarchical structure is specifically set as follows: The input preprocessing layer is the first layer of the network, receiving the preprocessed real-scene image; the output of the input preprocessing layer is directly connected to the input of the multi-level feature extraction layer, which contains three consecutive feature extraction sub-layers, with the number of convolutional kernels in each sub-layer gradually increasing to achieve a gradual increase in feature dimension; the outputs of the three feature extraction sub-layers are all connected to the input of the enhanced Ghost module, and the output of the enhanced Ghost module is connected to the input of the feature fusion layer. The feature fusion layer adopts a feature pyramid structure, and its output is connected to two parallel output branches, namely the feature point generation branch and the reliability map generation branch. The network training process is performed as follows: First, a large dataset of multi-view images of AR scenes is collected and labeled. The labeling includes the location of scene feature points in the images and the confidence level of feature matching. Then, the labeled dataset is divided into training set, validation set, and test set, with the division ratio set reasonably according to the data scale. The training set images are input into the lightweight convolutional neural network built in batches for forward propagation. Through the sequential processing of each layer module, the predicted scene feature points and feature reliability map are obtained. The feature matching loss between the prediction result and the labeled data is calculated, and the cross-entropy loss function is used to quantify the difference between the predicted value and the true value. A stochastic gradient descent optimizer is selected, and the parameters of each layer of the network, including convolutional kernel weights and bias terms, are updated by backpropagation based on the calculated loss value. During the training process, the network performance is evaluated using the validation set after a certain number of iterations, and the trend of the validation set loss is monitored. When the validation set loss remains stable for several consecutive rounds and no longer decreases, the network training is stopped, and the parameters of the trained network model are saved. The key parameters for network training are set as follows: the batch size is set to a fixed and reasonable size to ensure a balance between training efficiency and memory usage; the learning rate is dynamically adjusted using a cosine annealing strategy, with a relatively high learning rate set in the early stages of training to achieve rapid convergence, and the learning rate gradually reduced as training progresses to approach the optimal solution; the activation function is the ReLU function to enhance the network's ability to fit nonlinear features; the batch normalization parameter is set according to the usual network training settings to accelerate network convergence and suppress overfitting. In some embodiments, the connection between network layers can use partial skip connections instead of full connections. Skip connections are used to pass low-level features across layers, avoiding information loss of feature data during deep propagation, while still maintaining the main full-connected links, balancing the integrity of feature transmission and computational efficiency. Specifically, a skip connection can be set between the input preprocessing layer and the feature fusion layer, and skip connections can be set between adjacent sub-layers of the multi-level feature extraction layer, so that the raw features of the lower layers and the abstract features of the higher layers can interact directly, reducing feature attenuation during transmission, while not changing the overall lightweight design of the network, ensuring that the computational load does not increase significantly. S2.2. Extract low-level structural features of real-world scene images through an input preprocessing layer and a multi-level feature extraction layer: The real-world image processed by S1.1 is input into the input preprocessing layer of the lightweight convolutional neural network. The input preprocessing layer performs pixel value normalization on the image, mapping the grayscale or RGB values ​​of each pixel in the image to a preset uniform range, eliminating pixel value fluctuations caused by differences in lighting conditions in different images. At the same time, the image size is adjusted according to the input requirements of the lightweight convolutional neural network. The image is scaled to the input size specified by the network through an interpolation algorithm to ensure the matching of image size with network input channels. The preprocessed, standardized image is input into a multi-level feature extraction layer. The first-level feature extraction sub-layer uses a small-sized convolutional kernel to extract simple basic structural features such as edges and lines. The output feature map of the first-level sub-layer is input into the second-level feature extraction sub-layer, which uses a medium-sized convolutional kernel to further extract more complex low-level structural features such as texture features and local contour features based on edge features. The output feature map of the second-level sub-layer is input into the third-level feature extraction sub-layer, which uses multiple sets of convolutional operations with different kernel sizes to obtain low-level structural features at different resolutions, providing rich basic feature data for subsequent feature expansion and fusion. S2.3. The low-level structural features are expanded and optimized using the enhanced Ghost module to generate rich feature representations: The low-level structural features output from the multi-level feature extraction layer are concatenated along the channel dimension to integrate basic features of different resolutions and types, forming a unified feature input. The concatenated features are then input into the enhanced Ghost module. First, the low-level structural features are reorganized and aggregated through 1×1 convolution. The 1×1 convolution kernel learns and adjusts the weights to redistribute and fuse the channel information of the input features, generating a high-dimensional, dense intrinsic feature map with reduced dimensions and high information density. Next, feature expansion and redundancy suppression are performed on the intrinsic feature map by combining depthwise separable convolution with attention gating units: depthwise separable convolution splits the standard convolution into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operation on each input channel separately, while pointwise convolution is used to fuse the output features of depthwise convolution, which greatly reduces the amount of computation while maintaining the feature expressive power; attention gating units work in parallel with depthwise separable convolution, and strengthen the effective feature regions in the intrinsic feature map and suppress redundant information regions by calculating the importance weights of each region in the feature map, generating a ghost feature map that complements the intrinsic feature map; The intrinsic and ghost feature maps are fused element-wise with weights. During the fusion process, different weight coefficients are assigned according to the feature importance of the two types of feature maps to ensure sufficient preservation of effective features. The fused feature map is then processed by batch normalization to eliminate the distribution differences between different feature channels, making the feature data more suitable for subsequent network layer processing. Finally, the batch-normalized feature map is subjected to nonlinear transformation by an activation function to enhance the expressive power of the features and output a rich feature representation that is suitable for subsequent fusion. This process is embedded throughout the feature extraction link of a lightweight convolutional neural network. Through the synergistic effect of feature augmentation and redundancy suppression, the network's ability to capture scene detail features is enhanced. S2.4. Generate key feature points and feature reliability maps of the scene through the feature pyramid fusion layer and parallel output branches: The rich feature representation output by the enhanced Ghost module is input to the feature pyramid structure fusion layer. A bidirectional fusion strategy combining top-down and bottom-up approaches is adopted for multi-scale feature fusion. First, a top-down fusion process is performed, where the resolution of high-level semantic features is increased through upsampling to ensure that the resolution of high-level features is consistent with that of low-level features. Then, the upsampled high-level features are added and fused with the corresponding low-level features through each channel, so as to realize the guidance and supplementation of low-level structural features by high-level semantic features. Next, a bottom-up feedback adjustment process is performed, downsampling the fused features to compress the feature map size while retaining key feature information. The downsampled features are then passed to the previous fusion node through the feedback link, where they are fused again with the original features of that node to supplement detailed feature information, forming a bidirectional loop feature fusion mechanism. During the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of features at each scale. By learning the distribution pattern of scene features, the channel attention branch automatically identifies the feature channels that are effective for feature point extraction and assigns them higher weights, making the fused features more in line with the extraction requirements of key feature points in the scene. The final feature map after bidirectional fusion is input into two parallel output branches: the feature point generation branch performs feature filtering and localization on the fused features through continuous convolution and activation operations, and outputs a feature map representing the location of key feature points in the scene. The final coordinates of the key feature points in the scene are determined by threshold filtering; the reliability map generation branch outputs the matching confidence value corresponding to each feature point through a combination of convolutional layers and sigmoid activation function, forming a feature reliability map representing the confidence of feature matching. The confidence value is used to reflect the reliability of the matching between the key feature points in the scene and the feature points of the virtual object. S3. Integrate key feature points of the scene, IMU inertial measurement data, and virtual object 3D model data, correct feature matching deviations through geometric correction, and generate an initial alignment transformation matrix: S3.1. Filtering key feature points for high-confidence scenarios based on feature reliability maps: The feature reliability map generated by the parallel output branch of a lightweight convolutional neural network is read, and the confidence value corresponding to each pixel in the feature reliability map is extracted. The confidence value directly reflects the matching reliability of the feature point at that position. A confidence threshold is set to distinguish between valid feature points and low-reliability feature points. The key scene feature points corresponding to pixels with confidence values ​​higher than the threshold in the feature reliability map are selected, and low-reliability feature points with confidence values ​​lower than the threshold are removed to form a set of high-reliability feature points. During the selection process, the set of high-reliability feature points is deduplicated to remove duplicate feature point coordinates, ensuring that each feature point exists uniquely in the set, providing accurate feature input for subsequent feature matching. S3.2. Associate and match high-confidence feature points with virtual object feature points to determine the initial feature correspondence: Surface feature points are extracted from the 3D model data of virtual objects. The extraction process is based on the vertex coordinate data and face index data of virtual objects. Representative vertices on the surface of virtual objects are selected as virtual feature points, including edge vertices, corner vertices, and vertices in areas with obvious texture features. The extracted virtual feature points are classified according to coordinate information and feature type to establish a virtual feature point library. A feature distance matching algorithm is used to associate and match each feature point in the high-confidence feature point set with all virtual feature points in the virtual feature point library: the feature distance between each pair of feature points is calculated. The feature distance is calculated based on the multi-dimensional feature vector of the feature point, including grayscale features, texture features, spatial location features, etc. The smaller the feature distance, the higher the similarity between the two feature points. The matching pairs are sorted by similarity according to the feature distance calculation results. The scene feature point with the highest similarity and the virtual feature point are determined as the corresponding matching pair, forming the initial feature correspondence relationship. In some embodiments, the feature vector angle matching algorithm can be used instead of the feature distance matching algorithm. The angle between the feature vectors of scene feature points and virtual feature points is calculated. The smaller the angle, the higher the feature similarity. The initial feature correspondence is determined by sorting the feature vectors according to the size of the angle. In specific implementation, the feature vectors of scene feature points and virtual feature points are first normalized to eliminate the influence of vector length differences on the angle calculation. Then, the cosine value of the angle between each pair of feature vectors is calculated using the vector dot product formula. The closer the cosine value is to 1, the smaller the angle and the higher the similarity. The matching pairs are sorted in descending order of cosine value, and the first matching pair is selected as the initial feature correspondence. The two algorithms can be flexibly selected according to the actual feature type and data distribution, and neither of them requires the introduction of additional complex calculations, thus maintaining the efficiency of the matching process. S3.3. Incorporate IMU inertial measurement data to perform geometric correction on the initial feature correspondence: First, the IMU inertial measurement data undergoes further preprocessing, including timestamp calibration and zero-bias correction: timestamp calibration ensures that the IMU data is completely synchronized with the timestamps of the scene image and feature point data, and zero-bias correction eliminates the sensor's own zero-bias error by statistically analyzing the static output data of the IMU sensor; the preprocessed IMU data is then pre-integrated to accumulate triaxial acceleration and triaxial angular velocity data at adjacent time points, and to preliminarily estimate the motion attitude and position information of the AR device during the image acquisition process, thus obtaining the initial estimation results of the camera motion; A motion compensation model is constructed based on the initial estimation results of camera motion. This model calculates the positional offset of scene feature points caused by device motion during image acquisition based on the motion trajectory and posture changes of the AR device. The motion compensation model is then applied to the key scene feature points in the initial feature correspondence to compensate and adjust the position coordinates of the feature points, correcting the feature offset caused by device motion. By calculating the matching error between the compensated scene feature points and the corresponding virtual object 3D model feature points, the compensation parameters are iteratively corrected: after each iteration, the matching error is recalculated, and the parameters of the motion compensation model are adjusted according to the error change trend until the matching error stabilizes within a small range. Finally, the corrected feature correspondence is obtained, realizing the auxiliary correction role of IMU data for CNN feature matching and improving the accuracy of feature correspondence. S3.4. Calculate the spatial transformation parameters and generate the initial alignment transformation matrix: Based on the corrected feature correspondence, the two-dimensional coordinate information of the key feature points in the scene of each matching pair in the real scene image coordinate system and the three-dimensional coordinate information of the virtual feature points in the virtual object three-dimensional coordinate system are extracted respectively. The coordinate information in the virtual object three-dimensional coordinate system is transformed to the three-dimensional coordinate system corresponding to the real scene image through the coordinate transformation formula, so that the two types of coordinate data are in the same coordinate system, ensuring the effectiveness of the transformation parameter calculation. Based on the coordinates of scene feature points and virtual feature points in the same coordinate system, spatial transformation parameters are calculated. These parameters include translation, rotation, and scaling parameters: translation parameters characterize the positional offset of the virtual object relative to the real scene; rotation parameters describe the spatial rotation angle that the virtual object needs to be adjusted; and scaling parameters match the size ratio between the virtual object and the real scene. The calculated translation, rotation, and scaling parameters are then integrated according to matrix construction rules to generate an initial alignment transformation matrix. This matrix completely contains all the spatial transformation information required for aligning the virtual object with the real scene. S4. Adjust the virtual object's pose based on the initial alignment transformation matrix, and correct the matrix parameters using a temporal smoothing optimization algorithm to achieve real-time alignment between the real scene and the virtual object: S4.1. Adjust the preset pose parameters of the virtual object according to the initial alignment transformation matrix: The translation, rotation, and scaling parameters in the initial alignment transformation matrix are applied to adjust the preset posture parameters of the virtual object: the translation parameter directly acts on the initial position parameter of the virtual object, adjusting the spatial position of the virtual object in the AR scene according to the translation vector, so that the virtual object is initially moved to a position corresponding to the real scene; the rotation parameter adjusts the initial rotation angle parameter of the virtual object according to the transformation rules of the rotation matrix, correcting the spatial orientation of the virtual object, so that the posture of the virtual object is consistent with the perspective of the real scene; the scaling parameter is used to adjust the initial scaling ratio parameter of the virtual object, enlarging or shrinking the size of the virtual object, so that the size ratio of the virtual object is coordinated with that of objects in the real scene; through the coordinated adjustment of the three types of parameters, the position, angle, and size of the virtual object in three-dimensional space are updated, so that the virtual object is initially adapted to the spatial environment of the real scene; S4.2. Calculate the geometric error between the virtual object and the real scene, and quantify the degree of alignment: The edge contour features of the adjusted virtual object are extracted. The edge contour features are formed by connecting the surface edge point coordinates of the virtual object's 3D model. At the same time, the edge contour features of the corresponding region in the real scene image are extracted. The edge detection algorithm is used to identify the edge of the region in the real scene corresponding to the virtual object's position, forming the scene edge contour. The geometric error between two types of edge contour features is calculated, including multiple dimensions such as contour overlap and distance deviation. Contour overlap is characterized by the proportion of the overlapping area between the virtual object's edge contour and the real scene's edge contour; a higher overlap proportion indicates a better alignment effect. Distance deviation is quantified by calculating the average and maximum distances between corresponding points on the two types of contours; a smaller distance value indicates more accurate contour alignment. Through the comprehensive calculation of multi-dimensional geometric errors, the alignment degree between the virtual object and the real scene is fully quantified, providing an error basis for subsequent matrix correction. S4.3. Correct the initial alignment transformation matrix using a time-series smoothing optimization algorithm: The temporal smoothing optimization algorithm first extracts the changing trends of key scene feature points output by a lightweight convolutional neural network in multiple consecutive AR images. It then selects feature point data from adjacent frames as analysis samples, establishes a temporal correlation equation for the changes in feature point positions, determines temporal constraints, and constructs a temporal constraint model. The temporal constraint model is used to limit the amplitude of motion changes of virtual objects in consecutive frames to ensure the smoothness of motion trajectories. Based on the geometric error calculated in the current frame, the parameters of the initial alignment transformation matrix are iteratively adjusted: with the goal of minimizing the comprehensive error, which includes the geometric error and temporal constraint error of the current frame, the temporal constraint error is calculated by comparing the change in feature points between the current frame and historical frames; the iteration step size is dynamically adjusted, with a larger step size set in the early stage of iteration to quickly approach the optimal solution, and the step size is gradually reduced as the iteration progresses to improve the adjustment accuracy. During each iteration, the transformation matrix parameters of the current frame are corrected according to the changing trend of the comprehensive error, including fine-tuning of translation, rotation and scaling parameters. The iteration convergence condition is judged by the stability of CNN features. The change of key feature points of the scene obtained by two consecutive iterations is continuously monitored. When the change of feature is less than the preset range, it indicates that the matrix parameters have stabilized, the iteration stops and the corrected transformation matrix is ​​output. In some embodiments, the convergence condition can be determined by the change in comprehensive error. The iteration stops when the difference between the comprehensive errors of two consecutive iterations is less than a preset threshold. In specific implementation, the comprehensive error value of the previous iteration is recorded first, and the difference between the comprehensive error of the current iteration and the error of the previous iteration is calculated. When the difference remains below the preset threshold for multiple consecutive iterations, it is determined that the convergence condition is met, the iteration stops, and the corrected transformation matrix is ​​output. The two convergence judgment methods can be flexibly selected according to the error distribution and feature stability to ensure that the optimization process is always based on the CNN feature output and can achieve accurate correction of matrix parameters, avoiding the waste of computational resources caused by excessive iteration. S4.4. Incorporate spatiotemporal consistency constraints to output aligned AR images: The modified transformation matrix is ​​then applied again to adjust the pose of the virtual object, ensuring that the position, angle, and size of the virtual object are fully adapted to the real scene. Spatiotemporal consistency constraints are incorporated, which are based on the laws of physical motion and limit the motion trajectory of the virtual object in consecutive frames to avoid motion phenomena that do not conform to the laws of physics, such as abrupt displacement and jump rotation. The corrected virtual object posture is then checked for the final verification, including whether the geometric error meets the preset requirements, whether the virtual object's motion trajectory is smooth, and whether the virtual object blends naturally with the real scene. After the verification is passed, the adjusted virtual object is superimposed and synthesized with the real scene image to generate an aligned AR image. S4.5. Perform quality verification on the aligned AR image to form a closed-loop optimization chain: After outputting the aligned AR image, the quality of the AR image is verified. The verification process is completed through the feature consistency detection branch of a lightweight CNN: the feature consistency detection branch includes a feature extraction sub-layer and a consistency judgment sub-layer, which are connected by a fully connected method; the feature extraction sub-layer is used to extract the fusion features of virtual object features and real scene features in the aligned AR image, and the consistency judgment sub-layer is used to analyze the degree of consistency of the fusion features; The training process of the feature consistency detection branch is as follows: A large number of AR image samples, both fused and inconsistent, are collected as training data. After labeling, the samples are divided into training and validation sets. The training set samples are input into the branch network for training to learn the feature consistency discrimination rules. During training, the cross-entropy loss function is used to calculate the difference between the prediction results and the labels. The gradient descent optimizer is used to update the branch network parameters until the discrimination accuracy of the validation set stabilizes. Key parameters of the branch network include the convolution kernel size of the feature extraction sublayer, the feature dimension of the consistency judgment sublayer, and the activation parameters of the discrimination function. The feature consistency detection branch judges the fusion consistency between virtual object features and real scene features. When the consistency meets the preset requirements, the AR image is directly output. When the consistency does not meet the requirements, it is fed back to step S2 to adjust the feature extraction parameters of the lightweight convolutional neural network, including the convolution kernel weights of the multi-level feature extraction layer, the channel expansion coefficient of the enhanced Ghost module, and the weight allocation ratio of the feature fusion layer. The subsequent alignment process, such as feature extraction, feature matching, matrix calculation and pose adjustment, is re-performed to form a closed-loop optimization link with CNN as the core, ensuring that the final output AR image has a high-quality alignment effect. In some embodiments, during the closed-loop optimization process, key parameters of network training can be adjusted based on feedback results, including learning rate adjustment strategies, batch normalization parameter settings, etc., and the feature extraction capability of the network can be further optimized through retraining. Specifically, when feedback indicates insufficient feature extraction accuracy, the learning rate decay rate can be appropriately reduced to extend the network training cycle, allowing the network to learn scene feature patterns more fully. When feedback indicates feature redundancy, the channel expansion coefficient of the enhanced Ghost module can be adjusted to reduce the generation of redundant features, making feature extraction more targeted. This adjustment process does not require reconstructing the network structure; network performance can be improved simply through parameter optimization, ensuring the efficiency and feasibility of closed-loop optimization. This embodiment achieves real-time and accurate alignment of real-world scenes and virtual objects in AR scenarios through a systematic step design and the collaborative work of various modules. By constructing a lightweight convolutional neural network with an enhanced Ghost bottleneck structure, rationally designing the network's hierarchical structure and connection relationships, and optimizing training steps and key parameters, the completeness and accuracy of scene feature extraction are effectively improved while controlling the number of model parameters and computational load, providing a reliable feature foundation for the alignment process. By fusing IMU inertial measurement data to assist in feature matching correction, and constructing a motion compensation model to correct feature offsets caused by device motion, the impact of external factors on alignment accuracy is reduced, and the accuracy of feature correspondence is improved. Through time-series flattening... The combination of sliding optimization algorithm and spatiotemporal consistency constraints constructs a temporal constraint model and dynamically adjusts iteration parameters, ensuring the smoothness of the motion trajectory of virtual objects in consecutive frames and avoiding abrupt changes during the alignment process. The design of the closed-loop optimization link can promptly detect and correct deviations in the feature extraction and alignment process. Through parameter adjustment and process re-entry, the stability and reliability of the alignment effect are further improved, making the fusion of virtual objects and real scenes in the aligned AR image more natural. The overall method takes CNN feature extraction as the core, supplemented by multi-data fusion and multi-algorithm optimization, which significantly improves the accuracy and stability of AR scene image alignment while ensuring real-time performance, providing reliable technical support for the practical application of AR-related technologies.

[0039] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A lightweight CNN-based real-time AR scene alignment method, characterized in that, Includes the following steps: S1. Acquire multi-view real-world scene images of AR scenes, 3D model data of virtual objects, and IMU inertial measurement data; S2. Construct a lightweight convolutional neural network with an enhanced Ghost bottleneck structure. Extract low-level structural features and high-level semantic features from real scene images through this network in layers. After feature pyramid fusion, generate key feature points and feature reliability maps of the scene. S3. Taking the key feature points of the scene as the core, the system assists in fusing IMU inertial measurement data and virtual object 3D model data, corrects feature matching deviations through geometric correction, and generates an initial alignment transformation matrix. S4. Adjust the virtual object's pose based on the initial alignment transformation matrix, and correct the matrix parameters using a temporal smoothing optimization algorithm to achieve real-time alignment between the real scene and the virtual object.

2. The method according to claim 1, characterized in that, Step S1 includes: S1.

1. Collect multi-view real scene images from different directions and shooting angles around the core interactive area of ​​the AR scene. Keep the shooting device parameters consistent during the collection process, record the precise timestamp of each image collection moment, and perform preliminary noise filtering processing on the collected images. S1.

2. Obtain complete 3D model data of the virtual object, including vertex coordinate data, face index data, and texture mapping data. At the same time, obtain the preset posture parameters of the virtual object, including initial position parameters, initial rotation angle parameters, and initial scaling ratio parameters. S1.

3. Real-time acquisition of inertial measurement data from the built-in IMU sensor of the AR device. The IMU inertial measurement data includes three-axis acceleration data, three-axis angular velocity data, and three-axis attitude angle data. The acquired inertial measurement data is timestamped to ensure that it corresponds to the acquisition time of the real scene image.

3. The method according to claim 1, characterized in that, Step S2 includes: S2.

1. Construct a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer, and a parallel output branch. Data transmission links are established between each layer through a fully connected approach. The input preprocessing layer is used for image standardization, the multi-level feature extraction layer is used for extracting features of different dimensions in a hierarchical manner, the enhanced Ghost module is used for feature expansion and optimization, the feature fusion layer is used for multi-scale feature integration, and the parallel output branch is used to generate the target results respectively. S2.

2. The pixel values ​​of the real scene image are normalized by the input preprocessing layer, and the pixel values ​​are mapped to the preset range. At the same time, the image size is adjusted according to the input requirements of the lightweight convolutional neural network to ensure that the image size matches the network input channel. The processed image is input into the multi-level feature extraction layer, and the edge and texture low-level structure features at different resolutions are obtained through convolution operations with different kernel sizes. S2.

3. The low-level structural features output by the multi-level feature extraction layer are concatenated along the channel dimension and then input into the enhanced Ghost module. Through feature transformation and expansion operations within the module, a rich feature representation containing more detailed information is generated. S2.

4. By using a feature pyramid structure fusion layer, features at different scales are fused layer by layer. First, the high-level features are upsampled to make them consistent with the resolution of the low-level features. Then, the features are superimposed and fused. The fused features are input into parallel output branches. Through convolution and activation operations within the branches, key feature points of the scene and feature reliability maps representing the credibility of feature matching are generated respectively.

4. The method according to claim 1, characterized in that, Step S3 includes: S3.

1. Read the feature reliability map output by the lightweight convolutional neural network, extract the confidence value of each pixel in the feature reliability map, set a confidence threshold, and filter out the key feature points of the scene with confidence values ​​higher than the threshold to form a set of high confidence feature points. S3.

2. Extract surface feature points from the 3D model data of virtual objects, establish a virtual feature point library, use a feature distance matching algorithm to associate and match feature points in the high confidence feature point set with feature points in the virtual feature point library, calculate the similarity between pairs of feature points, and determine the initial feature correspondence based on the similarity ranking results. S3.

3. Auxiliary integration of pre-processed IMU inertial measurement data, calculation of the motion trajectory and attitude change of AR device based on IMU data, construction of motion offset model, geometric correction of feature point positions in the initial feature correspondence based on the model, and correction of feature offset caused by device motion; S3.

4. Based on the corrected feature correspondence, extract the coordinate information of the corresponding feature points in the real scene image coordinate system and the virtual object three-dimensional coordinate system, calculate the spatial transformation parameters through the coordinate transformation formula, the spatial transformation parameters include translation parameters, rotation parameters and scaling parameters, and integrate these parameters to generate an initial alignment transformation matrix.

5. The method according to claim 1, characterized in that, Step S4 includes: S4.

1. Apply the translation, rotation and scaling parameters in the initial alignment transformation matrix to the preset posture parameters of the virtual object, and update the position, angle and size of the virtual object in three-dimensional space according to the matrix transformation rules, so that the virtual object can initially adapt to the spatial environment of the real scene. S4.

2. Extract the edge contour features of the adjusted virtual object and the edge contour features of the corresponding region of the real scene image, calculate the overlap and distance deviation geometric error between the two types of edge contour features, and quantify the alignment degree between the virtual object and the real scene. S4.

3. The initial alignment transformation matrix is ​​corrected by using a temporal smoothing optimization algorithm. First, the historical alignment parameters of multiple consecutive frames are introduced as a reference. The difference between the alignment parameters of the current frame and the historical frames is calculated. An optimization objective function is constructed by combining the geometric error. The minimum value of the optimization objective function is obtained by iteratively solving the problem. S4.

4. Incorporate spatiotemporal consistency constraints to ensure that the motion trajectory of virtual objects in consecutive frames conforms to the laws of physical motion, avoiding abrupt displacements. Perform a final verification on the corrected virtual object posture, and output the aligned AR image after the verification passes.

6. The method according to claim 3, characterized in that, In step S2.3, the enhanced Ghost module first performs channel recombination and feature aggregation on low-level structural features through 1×1 convolution to generate a high-dimensional and dense intrinsic feature map; then, it performs feature expansion and redundancy suppression on the intrinsic feature map by combining depthwise separable convolution with attention gating units to generate a complementary Ghost feature map; after element-wise weighted fusion of the intrinsic feature map and the Ghost feature map, it outputs a rich feature representation adapted to subsequent fusion through batch normalization and activation function processing. This process embeds the feature extraction link of a lightweight convolutional neural network throughout, enhancing the network's ability to capture scene detail features.

7. The method according to claim 3, characterized in that, In step S2.4, the feature pyramid fusion layer adopts a bidirectional fusion strategy that combines top-down and bottom-up approaches. First, the high-level semantic features are upsampled and aligned with the low-level structural features, and then fused channel by channel. Next, the fused features are downsampled and adjusted to supplement detailed feature information. During the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of features at each scale, so that the fused features are more in line with the extraction requirements of key feature points in the scene.

8. The method according to claim 4, characterized in that, In step S3.3, the IMU inertial measurement data is first calibrated with timestamps and corrected for zero bias. An initial estimate of the camera motion is obtained through pre-integration. Then, a motion compensation model is constructed based on the initial estimate to compensate the position of key feature points of the scene output by the lightweight convolutional neural network. By calculating the matching error between the compensated feature points and the features of the 3D model of the virtual object, the compensation parameters are iteratively corrected to finally obtain the corrected feature correspondence, thereby realizing the auxiliary correction role of IMU data for CNN feature matching.

9. The method according to claim 5, characterized in that, In step S4.3, the temporal smoothing optimization algorithm first extracts the changing trend of key scene feature points output by the lightweight convolutional neural network in multiple consecutive frames and constructs a temporal constraint model; then, combined with the geometric error of the current frame, iteratively adjusts the parameters of the initial alignment transformation matrix; during the iteration process, the stability of the CNN features is used to determine the convergence condition of the iteration, and the iteration stops when the feature change is less than the preset range, ensuring that the optimization process is always based on the CNN feature output.

10. The method according to claim 1, characterized in that, After step S4, the quality of the aligned AR image is verified by using the feature consistency detection branch of a lightweight CNN to determine the fusion consistency between virtual object features and real scene features. When the consistency requirement is not met, feedback is sent to step S2 to adjust the feature extraction parameters of the lightweight convolutional neural network, and the feature extraction and subsequent alignment process are repeated to form a closed-loop optimization link with CNN as the core.

Images