An ar scene real-time alignment method of lightweight cnn

Abstract

The application discloses an AR scene real-time alignment method of a lightweight CNN and belongs to the field of augmented reality image processing.The method first acquires multi-view real images of an AR scene, three-dimensional model data of a virtual object and IMU inertial measurement data; then constructs a lightweight CNN containing an augmented Ghost bottleneck structure, hierarchically extracts low-level structure and high-level semantic features of the images, generates scene key feature points and a reliability map through feature pyramid fusion; fuses multi-class data with the feature points as the core, generates an initial alignment transformation matrix through geometric correction; finally adjusts the posture of the virtual object, performs time sequence smoothing optimization on the matrix, and completes real-time alignment.The method balances the lightweight of the model and the feature extraction capability, reduces the deviation caused by motion, improves the alignment accuracy and stability, guarantees the natural fusion of the virtual and real scenes, and adapts to the application requirements of mobile AR devices.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality image processing, in particular to a real-time alignment method of AR scene based on lightweight CNN. BACKGROUND

[0002] With the wide application of augmented reality (AR) technology in various scenes, real-time and accurate alignment of real scenes and virtual objects has become one of the core links to support the landing of AR technology. At present, the image alignment technology of AR scene mainly focuses on neural network, sensor data fusion, feature extraction and matching, etc. The industry generally pursues the real-time, accuracy and device adaptability of the alignment method. In the technical application level, convolutional neural network (CNN) has become the mainstream technology choice for feature processing in AR image alignment due to its strong feature extraction capability. At the same time, in order to adapt to the hardware resource limitation of mobile AR devices, the research and application of lightweight neural network model has become an important trend. In addition, the fusion strategy of inertial measurement unit (IMU) data and image data, virtual object three-dimensional model data, feature pyramid fusion, spatial transformation matrix calculation, time sequence optimization, etc. are also widely used in various alignment schemes, forming a technical framework based on feature extraction, core data fusion and supplementary optimization and correction, which promotes the development of AR alignment technology towards high efficiency and accuracy.

[0003] However, there are still many problems to be solved in the existing real-time image alignment technology of AR scene. The neural network model structure adopted by some alignment methods is not reasonable enough, which cannot effectively balance the model lightweight and feature extraction capability, resulting in either large model parameter quantity and high computational complexity, which is difficult to meet the real-time running demand of mobile AR devices, or simplifying the feature extraction process for the purpose of lightweight, which cannot fully capture the low-level structural features and high-level semantic features of the real scene, thereby affecting the extraction accuracy of the key feature points of the scene, leading to large alignment deviation. At the same time, the existing methods lack systematic design in the fusion application of multiple types of data, and cannot effectively coordinate the virtual object three-dimensional model data and IMU inertial measurement data with the scene feature points as the core. The feature matching process is easily affected by factors such as device motion, resulting in deviation, and lacks a targeted geometric correction mechanism. In addition, after obtaining the initial alignment transformation matrix, some methods do not perform effective time sequence smoothing optimization and correction, resulting in unstable motion trajectory of virtual objects in continuous frames, which is difficult to realize long-term accurate alignment of real scenes and virtual objects. These problems jointly restrict the application experience and popularization and landing of AR technology. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a real-time alignment method of AR scene based on lightweight CNN.

[0005] The purpose of the application is achieved by the following technical solutions:

[0006] An AR scene real-time alignment method of a lightweight CNN is provided, and the method comprises the following steps:

[0007] S1. Obtain multi-view real scene images of an AR scene, three-dimensional model data of a virtual object, and IMU inertial measurement data;

[0008] S2. Construct a lightweight convolutional neural network containing an enhanced Ghost bottleneck structure, extract low-level structural features and high-level semantic features of the real scene images through the network layer by layer, generate scene key feature points and feature reliability maps after feature pyramid fusion;

[0009] S3. Take the scene key feature points as the core, assist in fusing the IMU inertial measurement data and the three-dimensional model data of the virtual object, correct the feature matching deviation through geometric correction, and generate an initial alignment transformation matrix;

[0010] S4. Adjust the posture of the virtual object based on the initial alignment transformation matrix, correct the matrix parameters by combining a time sequence smoothing optimization algorithm, and complete the real-time alignment of the real scene and the virtual object.

[0011] Further, step S1 comprises:

[0012] S1.1. Around the core interactive area of the AR scene, collect multi-view real scene images at different positions and different shooting angles, keep the shooting device parameters consistent during the collection process, record the accurate time stamps of each image collection time, and perform preliminary noise filtering processing on the collected images;

[0013] S1.2. Obtain complete three-dimensional model data of the virtual object, wherein the three-dimensional model data includes vertex coordinate data, face index data, and texture mapping data, and obtain preset posture parameters of the virtual object, wherein the preset posture parameters include initial position parameters, initial rotation angle parameters, and initial scaling ratio parameters;

[0014] S1.3. Real-time collect inertial measurement data of the IMU sensor built-in the AR device, wherein the IMU inertial measurement data includes three-axis acceleration data, three-axis angular velocity data, and three-axis attitude angle data, and perform time stamp alignment processing on the collected inertial measurement data to ensure correspondence with the collection time of the real scene images.

[0015] Further, step S2 comprises:

[0016] S2.1. Build a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer, and a parallel output branch. The data transmission link between each layer is established through full connection. The input preprocessing layer is used for image standardization processing. The multi-level feature extraction layer is used for hierarchical extraction of different dimensional features. The enhanced Ghost module is used for feature expansion and optimization. The feature fusion layer is used for multi-scale feature integration. The parallel output branch is used to generate target results respectively;

[0017] S2.2. Perform pixel value normalization on real scene images through the input preprocessing layer to map the pixel values to a pre-set range. Adjust the image size according to the input requirements of the lightweight convolutional neural network to ensure that the image size matches the network input channel. Input the processed image into the multi-level feature extraction layer to obtain edge, texture, and low-level structure features at different resolutions through convolution operations with different kernel sizes.

[0018] S2.3. Concatenate the low-level structure features output by the multi-level feature extraction layer along the channel dimension and input them into the enhanced Ghost module. Through feature conversion and expansion operations within the module, generate rich feature representations containing more detailed information.

[0019] S2.4. Perform layer-by-layer fusion of features at different scales through the fusion layer of the feature pyramid structure. First, upsample the high-level features to match the resolution of the low-level features, then perform feature superposition fusion. The fused features are input into the parallel output branch, and through convolution and activation operations within the branch, generate scene key feature points and feature reliability maps representing feature matching confidence, respectively.

[0020] Further, step S3 includes:

[0021] S3.1. Read the feature reliability map output by the lightweight convolutional neural network, extract the confidence values of each pixel point in the feature reliability map, set a confidence threshold, and select scene key feature points with confidence values higher than the threshold to form a high-confidence feature point set.

[0022] S3.2. Extract surface feature points from the three-dimensional model data of virtual objects to establish a virtual feature point library. Use a feature distance matching algorithm to associate and match feature points in the high-confidence feature point set with feature points in the virtual feature point library, calculate the similarity of each pair of feature points, and determine the initial feature correspondence relationship according to the similarity sorting result.

[0023] S3.3. Auxiliary fusion of preprocessed IMU inertial measurement data, calculation of the motion trajectory and attitude change of the AR device based on the IMU data, construction of a motion offset model, geometric correction of the feature point positions in the initial feature correspondence relationship according to the model, and correction of the feature offset caused by device motion;

[0024] S3.4. Based on the corrected feature correspondence relationship, extract the coordinate information of the corresponding feature points in the real scene image coordinate system and the virtual object three-dimensional coordinate system, calculate the spatial transformation parameters including translation parameters, rotation parameters and scaling parameters through the coordinate conversion formula, and integrate these parameters to generate an initial alignment transformation matrix.

[0025] Further, step S4 includes:

[0026] S4.1. Apply the translation parameters, rotation parameters and scaling parameters in the initial alignment transformation matrix to the preset attitude parameter adjustment of the virtual object, update the position, angle and size of the virtual object in the three-dimensional space according to the matrix transformation rule, and make the virtual object preliminarily adapt to the space environment of the real scene;

[0027] S4.2. Extract the edge contour features of the adjusted virtual object and the edge contour features of the corresponding region of the real scene image, calculate the coincidence degree and distance deviation geometric error between the two types of edge contour features, and quantify the alignment degree of the virtual object and the real scene;

[0028] S4.3. Parameter correction of the initial alignment transformation matrix through a time sequence smoothing optimization algorithm, first introduce the historical alignment parameters of continuous multiple frames as a reference, calculate the difference between the current frame and the historical frame alignment parameters, combine the geometric error to construct an optimization objective function, and obtain the corrected alignment transformation matrix by iteratively solving the minimum value of the optimization objective function;

[0029] S4.4. Incorporate the spatiotemporal consistency constraint condition, constrain the motion trajectory of the virtual object in the continuous frames to comply with the physical motion law, avoid sudden displacement, and finally check the corrected virtual object attitude, and output the aligned AR image after the check is passed.

[0030] Further, in step S2.3, the enhanced Ghost module first reorganizes the channel and aggregates the low-level structural features through 1x1 convolution to generate high-dimensional intrinsic feature maps; then, the intrinsic feature maps are expanded and redundant suppressed through the combination of depth separable convolution and attention gate unit to generate complementary ghost feature maps; after the element-wise weighted fusion of the intrinsic feature maps and the ghost feature maps, the rich feature representation after the adaptation of the subsequent fusion is output through batch normalization and activation function processing, and the whole process is embedded in the feature extraction link of the lightweight convolutional neural network to strengthen the network's ability to capture scene detail features.

[0031] Further, in step S2.4, the feature pyramid fusion layer adopts a bidirectional fusion strategy combining top-down and bottom-up, first aligns the high-level semantic features through upsampling with the low-level structural features, and then performs channel-by-channel addition fusion; the fused features are then downsampled for feedback adjustment to supplement the detail feature information; during the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of each scale feature, making the fused features more suitable for the extraction needs of key scene feature points.

[0032] Further, in step S3.3, first, the IMU inertial measurement data is timestamped and corrected for zero offset, and the initial estimate of camera motion is obtained through pre-integration; then, based on the initial estimate, a motion compensation model is constructed to compensate for the position of the key scene feature points output by the lightweight convolutional neural network; by calculating the matching error between the compensated feature points and the three-dimensional model features of the virtual object, the compensation parameters are iteratively corrected, and finally the corrected feature correspondence is obtained, realizing the auxiliary correction of IMU data to CNN feature matching.

[0033] Further, in step S4.3, the time series smoothing optimization algorithm first extracts the trend of the key scene feature points output by the lightweight convolutional neural network in consecutive multiple frames to construct a time series constraint model; then, combined with the geometric error of the current frame, the initial alignment transformation matrix is iteratively adjusted; during the iteration process, the convergence condition is determined by the stability of the CNN features, and when the feature change is less than the preset range, the iteration is stopped to ensure that the optimization process is always based on the CNN feature output.

[0034] Further, after step S4, the quality of the aligned AR image is verified, and the consistency of the virtual object features and the real scene features is judged through the feature consistency detection branch of the lightweight CNN; when the consistency does not meet the requirements, feedback to step S2 is given to adjust the feature extraction parameters of the lightweight convolutional neural network, and the feature extraction and subsequent alignment process are performed again, forming a closed-loop optimization link centered on CNN.

[0035] The beneficial effects of the present application are:

[0036] (1) Light-weight CNN hierarchical feature extraction based on enhanced Ghost bottleneck structure, fusion of multiple types of data and time series smoothing optimization, real-time alignment of AR scene real and virtual objects, taking into account model lightweight and alignment accuracy;

[0037] (2) Feature pyramid fusion of multi-scale features, combined with geometric correction and IMU data assisted correction, to strengthen scene detail feature capture, effectively reduce feature matching deviation, and improve alignment benchmark accuracy;

[0038] (3) Spatiotemporal consistency constraint and closed-loop optimization link synergy, ensuring smooth motion of virtual objects in consecutive frames, avoiding sudden displacement, and further improving alignment stability and virtual scene fusion naturalness. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A light-weight CNN AR scene real-time alignment method step flowchart is provided.

[0040] Figure 2 A light-weight CNN AR scene real-time image alignment method specific implementation step flowchart is provided for an embodiment. DETAILED DESCRIPTION

[0041] The technical solutions of the present application will be described in detail below in conjunction with embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0042] Embodiment 1

[0043] Referring to Figure 1 , a light-weight CNN AR scene real-time alignment method is provided, comprising the following steps:

[0044] S1. Obtain multi-view real scene images of AR scene, three-dimensional model data of virtual objects and IMU inertial measurement data;

[0045] S2. Construct a light-weight convolutional neural network containing an enhanced Ghost bottleneck structure, extract low-level structural features and high-level semantic features of real scene images through the network, and generate scene key feature points and feature reliability map after feature pyramid fusion;

[0046] S3. Take the scene key feature points as the core, auxiliary fusion of IMU inertial measurement data and virtual object three-dimensional model data, correct the feature matching deviation through geometric correction, and generate an initial alignment transformation matrix;

[0047] S4. Adjusting the virtual object pose based on the initial alignment transformation matrix, correcting the matrix parameters by combining the time sequence smoothing optimization algorithm, and completing the real-time alignment of the real scene and the virtual object.

[0048] In some embodiments, step S3 can first fuse the IMU inertial measurement data and the virtual object three-dimensional model data to generate a preliminary pose reference of the virtual object, and then perform association matching with the scene key feature points. Through bidirectional correction, the feature matching deviation and the virtual object pose deviation are corrected, and finally the initial alignment transformation matrix is generated. The overall process improves the accuracy of the alignment reference through bidirectional data interaction.

[0049] Step S1 includes:

[0050] S1.1. Around the core interaction area of the AR scene, collect multi-view real scene images at different orientations and different shooting angles. Keep the shooting device parameters consistent during the collection process, record the accurate time stamp of each image collection time, and perform preliminary noise filtering processing on the collected images;

[0051] S1.2. Obtain complete three-dimensional model data of the virtual object, which includes vertex coordinate data, face index data, and texture mapping data. Also obtain the preset pose parameters of the virtual object, which include initial position parameters, initial rotation angle parameters, and initial scaling ratio parameters;

[0052] S1.3. Real-time collection of inertial measurement data of the built-in IMU sensor of the AR device, which includes three-axis acceleration data, three-axis angular velocity data, and three-axis attitude angle data. Perform time stamp alignment processing on the collected inertial measurement data to ensure correspondence with the collection time of the real scene images.

[0053] In some embodiments, step S1.1 can adopt a dynamic view angle collection strategy, adjust the collection angle and orientation according to the real-time changes of the AR scene, and replace single frame noise filtering processing with multi-frame superposition denoising after collection; step S1.2 can additionally obtain physical attribute data of the virtual object for physical constraints in subsequent pose adjustment; step S1.3 can perform sliding window filtering processing on the IMU inertial measurement data to further eliminate abnormal data and improve data stability.

[0054] Step S2 includes:

[0055] S2.1. Build a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer, and a parallel output branch. Each layer is connected by a full connection method to establish a data transmission link. The input preprocessing layer is used for image standardization processing. The multi-level feature extraction layer is used for hierarchical extraction of different dimensional features. The enhanced Ghost module is used for feature expansion and optimization. The feature fusion layer is used for multi-scale feature integration. The parallel output branch is used to generate target results respectively;

[0056] S2.2. Perform pixel value normalization on real scene images through the input preprocessing layer to map the pixel values to a pre-set range. Adjust the image size according to the input requirements of the lightweight convolutional neural network to ensure that the image size matches the network input channel. Input the processed image into the multi-level feature extraction layer to obtain edge, texture, and low-level structure features at different resolutions through convolution operations with different kernel sizes.

[0057] S2.3. Concatenate the low-level structure features output by the multi-level feature extraction layer along the channel dimension and input them into the enhanced Ghost module. Through feature conversion and expansion operations within the module, generate rich feature representations containing more detailed information.

[0058] S2.4. Perform layer-by-layer fusion of features at different scales through the fusion layer with a feature pyramid structure. First, upsample the high-level features to match the resolution of the low-level features, and then perform feature superposition fusion. The fused features are input into the parallel output branch to generate scene key feature points and feature reliability maps representing feature matching confidence through convolution and activation operations within the branch.

[0059] In some embodiments, step S2.1 can use a hybrid link of "full connection + local jump connection". Jump connections are set between the input preprocessing layer and the feature fusion layer, and between adjacent sub-layers of the multi-level feature extraction layer to reduce feature transmission loss. Step S2.3 can replace the enhanced Ghost module with a lightweight attention enhancement module to strengthen effective features through the synergistic effect of channel attention and spatial attention. Step S2.4 can use attention weighted fusion instead of feature superposition fusion to assign dynamic weights according to the importance of features at different scales.

[0060] Step S3 includes:

[0061] S3.1. Read the feature reliability map output by the lightweight convolutional neural network, extract the confidence values of each pixel point in the feature reliability map, set a confidence threshold, and select scene key feature points with confidence values higher than the threshold to form a high-confidence feature point set.

[0062] S3.2. Extract surface feature points from the three-dimensional model data of the virtual object, establish a virtual feature point library, and use a feature distance matching algorithm to associate and match the feature points in the high-confidence feature point set with the feature points in the virtual feature point library, calculate the similarity of each pair of feature points, and determine the initial feature correspondence relationship according to the similarity sorting result;

[0063] S3.3. Assist in integrating the preprocessed IMU inertial measurement data, calculate the motion trajectory and attitude change of the AR device based on the IMU data, construct a motion offset model, and perform geometric correction on the feature point positions in the initial feature correspondence relationship according to the model to correct the feature offset caused by device motion;

[0064] S3.4. Based on the corrected feature correspondence relationship, extract the coordinate information of the corresponding feature points in the real scene image coordinate system and the virtual object three-dimensional coordinate system, calculate the spatial transformation parameters including translation parameters, rotation parameters and scaling parameters through the coordinate conversion formula, and integrate these parameters to generate an initial alignment transformation matrix.

[0065] In some embodiments, step S3.2 can use a feature vector angle matching algorithm instead of a feature distance matching algorithm to determine the similarity by calculating the cosine value of the angle between the feature vectors; step S3.3 can construct an extended Kalman filter model to tightly couple and fuse the IMU data and the feature point position data, and simultaneously complete motion estimation and geometric correction; step S3.4 can solve the spatial transformation parameters by least squares method, and combine robust estimation to eliminate the influence of abnormal corresponding point pairs on parameter calculation.

[0066] Step S4 includes:

[0067] S4.1. Apply the translation parameters, rotation parameters and scaling parameters in the initial alignment transformation matrix to the preset attitude parameter adjustment of the virtual object, update the position, angle and size of the virtual object in the three-dimensional space according to the matrix transformation rules, and make the virtual object preliminarily adapt to the spatial environment of the real scene;

[0068] S4.2. Extract the edge contour features of the adjusted virtual object and the edge contour features of the corresponding area of the real scene image, calculate the coincidence degree and distance deviation geometric error between the two types of edge contour features, and quantify the alignment degree of the virtual object and the real scene;

[0069] S4.3. Perform parameter correction on the initial alignment transformation matrix through a time sequence smoothing optimization algorithm, first introduce the historical alignment parameters of continuous multiple frames as a reference, calculate the difference between the current frame and the historical frame alignment parameters, combine the geometric error to construct an optimization objective function, and obtain the corrected alignment transformation matrix by iteratively solving the minimum value of the optimization objective function;

[0070] S4.4. Incorporate the spatiotemporal consistency constraint to constrain the motion trajectory of the virtual object in the continuous frame to comply with the physical motion law, avoid sudden displacement, and finally check the corrected virtual object pose. If the check is passed, output the aligned AR image.

[0071] In some embodiments, step S4.2 can increase the photometric error calculation to supplement the quantitative alignment degree by comparing the pixel brightness distribution difference between the virtual object and the corresponding area of the real scene; step S4.3 can use a gradient descent algorithm instead of a temporal smoothing optimization algorithm to directly minimize the geometric error as the iteration target to correct the matrix parameters; and step S4.4 can incorporate the photometric consistency constraint to ensure that the lighting effect of the virtual object matches that of the real scene, thereby improving the naturalness of the fusion.

[0072] In step S2.3, the enhanced Ghost module first reorganizes and aggregates the low-level structural features through 1x1 convolution to generate high-dimensional intrinsic feature maps; then it expands and suppresses the intrinsic feature maps through a combination of depth separable convolution and attention gate units to generate complementary ghost feature maps; after element-wise weighted fusion of the intrinsic feature maps and the ghost feature maps, the process is processed through batch normalization and activation function to output rich feature representations adapted for subsequent fusion. This process is embedded in the feature extraction link of the lightweight convolutional neural network throughout, strengthening the network's ability to capture scene detail features.

[0073] In some embodiments, the enhanced Ghost module can first split and locally aggregate the low-level structural features through grouped convolution to generate multiple groups of low-dimensional intrinsic feature maps; then it can expand and suppress each group of intrinsic feature maps through a combination of point convolution and spatial attention units to generate multiple groups of ghost feature maps; after channel concatenation of each group of intrinsic feature maps and the corresponding ghost feature maps, the global attention module is used to filter key feature channels, and then the batch normalization and activation function are used for processing to output rich feature representations, further improving the relevance and effectiveness of the features.

[0074] In step S2.4, the feature pyramid fusion layer adopts a bidirectional fusion strategy combining top-down and bottom-up, first aligns the high-level semantic features with the low-level structural features through upsampling, and then performs channel-by-channel addition fusion; then the fused features are fed back for adjustment through downsampling to supplement the detail feature information; during the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of each scale feature, making the fused features more suitable for the extraction needs of key feature points in the scene.

[0075] In some embodiments, the feature pyramid fusion layer can adopt a bottom-up and top-down reverse fusion order, first align the low-level structural features with the high-level semantic features through downsampling, and perform element-by-element multiplication fusion to highlight key features; then perform upsampling feedback adjustment on the fused features to supplement global feature information; dynamically allocate the weights of the features of each region through the spatial attention branch during the fusion process, focus on the feature fusion of the key areas of the scene, and make the generated scene key feature point positioning more accurate.

[0076] In step S3.3, the IMU inertial measurement data is first timestamped and corrected for zero offset, and the initial estimate of camera motion is obtained by pre-integration; then, based on the initial estimate, a motion compensation model is constructed to compensate for the position of the scene key feature points output by the lightweight convolutional neural network; by calculating the matching error between the compensated feature points and the features of the virtual object three-dimensional model, the compensation parameters are iteratively corrected, and finally the corrected feature correspondence is obtained, realizing the auxiliary correction of the CNN feature matching by the IMU data.

[0077] In some embodiments, the IMU inertial measurement data and the timestamp of the real scene image can be first calibrated to sub-millisecond level, and the continuous estimate of camera motion can be obtained by pre-integration through a sliding window; then, based on the continuous estimate, an adaptive motion compensation model is constructed, the compensation coefficients are dynamically adjusted according to the spatial position of the feature points, and the scene key feature points are compensated in different regions; by calculating the Euclidean distance error and cosine similarity error between the compensated feature points and the features of the virtual object three-dimensional model, the compensation parameters are corrected by weighted iteration, further improving the accuracy of the feature correspondence.

[0078] In step S4.3, the time series smoothing optimization algorithm first extracts the change trend of the scene key feature points output by the lightweight convolutional neural network in continuous multiple frames, and constructs a time series constraint model; then, combined with the geometric error of the current frame, the initial alignment transformation matrix is iteratively adjusted; during the iteration process, the convergence condition is judged by the stability of the CNN features, and the iteration is stopped when the feature change is less than the preset range, ensuring that the optimization process is always based on the CNN feature output.

[0079] In some embodiments, the time series smoothing optimization algorithm can first extract the change trend of the alignment parameters in continuous multiple frames within a sliding window, construct a time series constraint model based on time weight, and the weight of the alignment parameters of recent frames is higher than that of distant frames; then, combined with the geometric error and photometric error of the current frame, a multi-objective optimization objective function is constructed; when iteratively adjusting the parameters of the initial alignment transformation matrix, the convergence condition is judged by the change of the optimization objective function, and the iteration is stopped when the difference of the objective function of the last two iterations is less than the preset range, taking into account the optimization efficiency and alignment accuracy.

[0080] After step S4, the aligned AR image is verified for quality, the consistency of the virtual object features and the real scene features is judged by the feature consistency detection branch of the lightweight CNN, and when the consistency does not meet the requirements, feedback is given to step S2, the feature extraction parameters of the lightweight convolutional neural network are adjusted, and the feature extraction and subsequent alignment process are performed again to form a closed-loop optimization link with CNN as the core.

[0081] In some embodiments, when the quality of the aligned AR image is verified, the feature consistency detection branch and the pixel consistency detection branch of the lightweight CNN can work together, the feature consistency detection branch judges the fusion of high-level semantic features, and the pixel consistency detection branch judges the matching of low-level pixel distribution; when either branch detects that the consistency does not meet the requirements, feedback is given to steps S2 and S3, and the feature extraction parameters of the lightweight convolutional neural network and the parameters of the feature matching algorithm are adjusted, and the feature extraction, matching and subsequent alignment process are performed again to form a more comprehensive closed-loop optimization link and further improve the alignment stability.

[0082] Embodiment 2

[0083] The embodiment provides a specific implementation process of a lightweight CNN AR scene real-time image alignment method, which realizes the accurate real-time alignment of real scenes and virtual objects through systematic data acquisition, feature extraction, data fusion and optimization adjustment, as shown in Figure 2 The specific step implementation process is as follows:

[0084] S1. Obtain multi-view real scene images of an AR scene, three-dimensional model data of a virtual object, and IMU inertial measurement data:

[0085] S1.1. Around the core interaction area of the AR scene, collect multi-view real scene images at different orientations and different shooting angles:

[0086] During the collection process, keep the optical parameters and imaging modes of the shooting device consistent to avoid non-scene differences in image features caused by changes in device parameters; record the accurate time stamp of each image collection time, which needs to be accurate to the instant of image collection to ensure the time synchronization during subsequent data fusion; at the same time, perform preliminary noise filtering processing on the collected images, remove random noise generated during the collection process due to environmental interference and device jitter through basic image smoothing operation, and retain the original structure and feature information of the scene in the image;

[0087] S1.2. Obtain complete three-dimensional model data and preset posture parameters of the virtual object:

[0088] The acquired virtual object three-dimensional model data includes vertex coordinate data, face index data and texture mapping data, wherein the vertex coordinate data is used to represent the three-dimensional space structure of the virtual object, the face index data is used to define the connection relationship between the vertices to constitute the surface morphology of the virtual object, and the texture mapping data is used to present the surface texture details of the virtual object; meanwhile, preset posture parameters of the virtual object are acquired, the preset posture parameters including initial position parameters, initial rotation angle parameters and initial scaling ratio parameters, the initial position parameters being used to set the initial placement position of the virtual object in the AR scene, the initial rotation angle parameters being used to define the spatial orientation of the virtual object in the initial state, and the initial scaling ratio parameters being used to match the size ratio relationship between the virtual object and the real scene;

[0089] S1.3. Real-time acquisition and preprocessing of inertial measurement data of the built-in IMU sensor of the AR device:

[0090] The inertial measurement data output by the built-in IMU sensor of the AR device is acquired in real time, the IMU inertial measurement data including three-axis acceleration data, three-axis angular velocity data and three-axis attitude angle data, the three-axis acceleration data being used to reflect the linear motion state of the AR device, the three-axis angular velocity data being used to represent the rotational motion state of the AR device, and the three-axis attitude angle data being used to describe the attitude information of the AR device in space; the acquired inertial measurement data is subjected to timestamp alignment processing, so that the timestamps of the IMU data are one-to-one corresponding to the acquisition time of the real scene image, ensuring the synchronization of the two types of data in the time dimension; at the same time, the IMU data is subjected to preliminary outlier rejection processing, removing abnormal data points generated due to instantaneous sensor failure or external severe interference, ensuring the reliability of the inertial measurement data;

[0091] S2. Constructing a lightweight convolutional neural network containing an enhanced Ghost bottleneck structure, extracting low-level structure features and high-level semantic features of the real scene image through the network layer by layer, and generating scene key feature points and feature reliability maps after feature pyramid fusion:

[0092] S2.1. Building a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer and a parallel output branch and completing training:

[0093] When building a lightweight convolutional neural network, a full connection method is used to establish a data transmission link between layers to ensure effective transmission of feature data between modules. The input preprocessing layer is specially used for image standardization processing to provide uniform specification data input for subsequent feature extraction. The multi-level feature extraction layer is used to extract image features of different dimensions in layers, gradually progressing from basic structural features to complex semantic features. The enhanced Ghost module is used for feature expansion and optimization to control the parameter amount while improving the feature expression capability. The feature fusion layer is used for multi-scale feature integration to aggregate effective features at different resolutions. The parallel output branch is used to generate two types of target results, scene key feature points and feature reliability maps.

[0094] The hierarchical structure of the network is specifically set as follows: the input preprocessing layer is the first layer of the network, which receives the real scene images after preliminary processing. The output end of the input preprocessing layer is directly connected to the input end of the multi-level feature extraction layer. The multi-level feature extraction layer includes three consecutive feature extraction sub-layers, and the number of convolution kernels of each level of sub-layer gradually increases to realize the gradual improvement of feature dimension. The output ends of the three-level feature extraction sub-layers are connected to the input end of the enhanced Ghost module. The output end of the enhanced Ghost module is connected to the input end of the feature fusion layer. The feature fusion layer adopts a feature pyramid structure, and its output end is connected to two parallel output branches, i.e., the feature point generation branch and the reliability map generation branch.

[0095] The network training process is performed according to the following steps: first, a large amount of multi-view image dataset of AR scenes is collected, and the dataset is labeled, including the positions of scene feature points in the image and the confidence level of feature matching. Then, the labeled dataset is divided into a training set, a validation set and a test set, and the division ratio is reasonably set according to the data size. The training set images are input into the built lightweight convolutional neural network for forward propagation in batches. Through the sequential processing of each layer module, the predicted scene feature points and feature reliability maps are obtained. The feature matching loss between the predicted results and the labeled data is calculated, and the cross-entropy loss function is used to quantify the difference between the predicted value and the true value. The stochastic gradient descent optimizer is selected, and the parameters of each layer of the network are updated according to the calculated loss value, including the convolution kernel weight, the bias term, etc. During the training process, the network performance is evaluated using the validation set after a certain number of iterations, and the trend of the validation set loss is monitored. When the validation set loss remains stable for several rounds and no longer decreases, the network training is stopped, and the trained network model parameters are saved.

[0096] The key parameter settings of network training are as follows: the batch size is set to a reasonable fixed size to ensure the balance between training efficiency and memory occupation; the learning rate is dynamically adjusted by using the cosine annealing strategy, a relatively high learning rate is set in the initial training stage to quickly converge, and the learning rate is gradually reduced to approach the optimal solution as the training process progresses; the activation function uses the ReLU function to enhance the network's fitting ability to nonlinear features; the batch normalization parameters are set according to the conventional network training, which is used to accelerate network convergence and suppress overfitting;

[0097] In some embodiments, the connection mode between the layers of the network can use partial jump connection instead of full connection. The jump connection is used to transfer low-level features across layers to avoid information loss during deep layer propagation, while still maintaining the main full connection link, taking into account the integrity and computational efficiency of feature transmission. Specifically, a jump connection can be set between the input preprocessing layer and the feature fusion layer, and a jump connection can be set between adjacent sub-layers of the multi-level feature extraction layer, so that the original features of the low-level and the abstract features of the high-level can directly interact, reducing the attenuation of the features in the transmission process, while not changing the overall lightweight design of the network, ensuring that the computational complexity does not increase significantly.

[0098] S2.2. Extract low-level structural features of real scene images through the input preprocessing layer and the multi-level feature extraction layer:

[0099] The real scene image processed by S1.1 is input into the input preprocessing layer of the lightweight convolutional neural network. The input preprocessing layer performs pixel value normalization processing on the image, mapping the gray value or RGB value of each pixel in the image to a pre-set unified range, eliminating the pixel value fluctuations caused by differences in lighting conditions between different images. At the same time, the image size is adjusted according to the input requirements of the lightweight convolutional neural network. The image is scaled to the specified input size of the network through an interpolation algorithm to ensure the matching of the image size and the network input channel.

[0100] The preprocessed standardized image is input into the multi-level feature extraction layer. The first level feature extraction sub-layer uses a smaller size convolution kernel for convolution operation, focusing on extracting basic structural features such as edges and lines in the image. The output feature map of the first level sub-layer is input into the second level feature extraction sub-layer, which uses a medium size convolution kernel to further extract more complex low-level structural features such as texture features and local contour features based on edge features. The output feature map of the second level sub-layer is input into the third level feature extraction sub-layer, which obtains low-level structural features at different resolutions through multiple convolution operations with different kernel sizes, providing rich basic feature data for subsequent feature expansion and fusion.

[0101] S2.3. Expand and optimize low-level structural features through the enhanced Ghost module to generate rich feature representations:

[0102] The low-level structure features at each level output by the multi-level feature extraction layer are spliced in the channel dimension, integrating different resolutions and different types of basic features to form a unified dimension feature input; the spliced feature input is input into the enhanced Ghost module, which first reorganizes and aggregates the low-level structure features through 1x1 convolution. The 1x1 convolution kernel adjusts the weights through learning to redistribute and fuse the channel information of the input features, generating high-dimensional dense intrinsic feature maps with reduced dimension and high information density;

[0103] Then, the intrinsic feature maps are expanded and redundant information is suppressed through a combination of depth separable convolution and attention gate units: the depth separable convolution separates the standard convolution into two steps, depth convolution and pointwise convolution. The depth convolution performs convolution operation on each input channel separately, and the pointwise convolution is used to fuse the output features of the depth convolution, significantly reducing the computational complexity while maintaining the feature expression capability. The attention gate unit works in parallel with the depth separable convolution, and by calculating the importance weight of each region of the feature map, it strengthens the effective feature regions and suppresses the redundant information regions in the intrinsic feature map, generating a ghost feature map complementary to the intrinsic feature map.

[0104] The intrinsic feature maps and ghost feature maps are element-wise weighted fused, and different weight coefficients are assigned according to the feature importance of the two types of feature maps during the fusion process to ensure the full retention of effective features. The fused feature map is processed by batch normalization to eliminate the distribution difference between different feature channels, making the feature data more suitable for subsequent network layer processing. Finally, the batch-normalized feature map is nonlinearly transformed by an activation function to enhance the expression capability of the feature, outputting a rich feature representation suitable for subsequent fusion. This process is embedded in the feature extraction link of the lightweight convolutional neural network throughout, and through the synergistic effect of feature expansion and redundancy suppression, the network's ability to capture scene detail features is strengthened.

[0105] S2.4. Generate scene key feature points and feature reliability map through feature pyramid fusion layer and parallel output branch:

[0106] The rich feature representation output by the enhanced Ghost module is input into the fusion layer of the feature pyramid structure, which adopts a bidirectional fusion strategy combining top-down and bottom-up to fuse multi-scale features. First, the top-down fusion process is performed, which raises the resolution of high-level semantic features through upsampling operation, making the resolution of high-level features consistent with that of low-level features. Then, the upsampled high-level features are added to the corresponding low-level features in each channel to realize the guidance and supplementation of high-level semantic features to low-level structure features.

[0107] Then a bottom-up feedback adjustment process is performed, the fused features are down-sampled to compress the feature map size while retaining key feature information, and the down-sampled features are transmitted to the previous fusion node through a feedback link to fuse with the original features of the node again to supplement the detailed feature information and form a bidirectional cyclic feature fusion mechanism; during the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of the features of each scale, the channel attention branch automatically identifies the feature channels effective for feature point extraction by learning the distribution law of scene features, gives higher weights to the feature channels, and makes the fused features more suitable for the extraction requirements of scene key feature points;

[0108] The final feature map after bidirectional fusion is input into two parallel output branches: the feature point generation branch performs feature screening and positioning on the fused features through continuous convolution and activation operations, outputs a feature map representing the positions of scene key feature points, and determines the final scene key feature point coordinates through threshold screening; the reliability map generation branch outputs the matching confidence values of each feature point through the combination of a convolution layer and a sigmoid activation function, forms a feature reliability map representing the feature matching confidence, and the confidence values are used to reflect the reliability of the matching of scene key feature points and virtual object feature points;

[0109] S3. Fusion of scene key feature points, IMU inertial measurement data and virtual object three-dimensional model data, correction of feature matching deviation through geometric correction, and generation of an initial alignment transformation matrix:

[0110] S3.1. Screening of high-confidence scene key feature points based on the feature reliability map:

[0111] The feature reliability map generated by the parallel output branch of the lightweight convolutional neural network is read, and the confidence value corresponding to each pixel point in the feature reliability map is extracted. The confidence value directly reflects the matching reliability of the feature point at that position. A confidence threshold is set to distinguish between effective feature points and low-reliability feature points. The scene key feature points corresponding to the pixel points in the feature reliability map with confidence values higher than the threshold are screened out, and the low-reliability feature points with confidence values lower than the threshold are removed to form a high-confidence feature point set. During the screening process, the high-confidence feature point set is de-duplicated to remove duplicate feature point coordinates, ensuring that each feature point exists uniquely in the set and providing accurate feature input for subsequent feature matching;

[0112] S3.2. Association and matching of high-confidence feature points and virtual object feature points to determine the initial feature correspondence:

[0113] Surface feature points are extracted from the three-dimensional model data of the virtual object, the extraction process is based on vertex coordinate data and face index data of the virtual object, representative vertices on the surface of the virtual object are selected as virtual feature points, including edge vertices, corner vertices, and vertices in regions with obvious texture features, etc., the extracted virtual feature points are classified according to coordinate information and feature types, and a virtual feature point library is established;

[0114] A feature distance matching algorithm is used to associate and match each feature point in the high-confidence feature point set with all virtual feature points in the virtual feature point library: the feature distance between each pair of feature points is calculated, the calculation of the feature distance is based on the multi-dimensional feature vector of the feature point, including the gray feature, the texture feature, the spatial position feature, etc., and the smaller the feature distance is, the higher the similarity of the two feature points is; the matching pairs are sorted according to the feature distance calculation result, the scene feature point with the highest similarity and the virtual feature point are determined as the matching pair corresponding to each other, and an initial feature correspondence relationship is formed;

[0115] In some embodiments, a feature vector angle matching algorithm can be used instead of the feature distance matching algorithm, the angle between the feature vectors of the scene feature points and the virtual feature points is calculated, the smaller the angle is, the higher the feature similarity is, and the initial feature correspondence relationship is determined according to the angle size sorting; in specific implementation, the feature vectors of the scene feature points and the virtual feature points are first normalized to eliminate the influence of the vector length difference on the angle calculation, and then the cosine value of the angle between each pair of feature vectors is calculated through the vector dot product formula, the closer the cosine value is to 1, the smaller the angle is and the higher the similarity is, the matching pairs are sorted in descending order of the cosine value, and the matching pairs at the top of the sorting are selected as the initial feature correspondence relationship, the two algorithms can be flexibly selected according to the actual feature type and data distribution, and both do not need to introduce additional complex calculation, maintaining the efficiency of the matching process;

[0116] S3.3. Fusing IMU inertial measurement data, geometric correction is performed on the initial feature correspondence relationship:

[0117] First, the IMU inertial measurement data is further preprocessed, including timestamp calibration and zero offset correction: the timestamp calibration ensures that the IMU data is completely synchronized with the timestamps of the scene image and the feature point data, and the zero offset correction eliminates the zero offset error of the sensor itself by statistical analysis of the static output data of the IMU sensor; the pre-processed IMU data is pre-integrated, the three-axis acceleration data and the three-axis angular velocity data of adjacent time points are accumulated, the motion posture and position information of the AR device in the image acquisition process are preliminarily estimated, and the initial estimation result of the camera motion is obtained;

[0118] A motion compensation model is constructed based on the initial estimation result of camera motion, which calculates the position offset of the scene feature points caused by device motion during image acquisition according to the motion trajectory and attitude change of the AR device; the motion compensation model is applied to the scene key feature points in the initial feature correspondence relationship, and the position coordinates of the feature points are compensated and adjusted to correct the feature offset caused by device motion;

[0119] By calculating the matching error between the compensated scene feature points and the corresponding virtual object three-dimensional model feature points, the compensation parameters are iteratively corrected: the matching error is recalculated after each iteration, the parameters of the motion compensation model are adjusted according to the error change trend, and the matching error is stable in a small range, and finally the corrected feature correspondence relationship is obtained, which realizes the auxiliary correction of IMU data to CNN feature matching and improves the accuracy of the feature correspondence relationship;

[0120] S3.4. Calculate the spatial transformation parameters to generate an initial alignment transformation matrix:

[0121] Based on the corrected feature correspondence relationship, the two-dimensional coordinate information of the scene key feature points in the real scene image coordinate system and the three-dimensional coordinate information of the virtual feature points in the virtual object three-dimensional coordinate system are extracted respectively in each matching pair; through the coordinate conversion formula, the coordinate information in the virtual object three-dimensional coordinate system is converted to the three-dimensional coordinate system corresponding to the real scene image, so that the two types of coordinate data are in the same coordinate system, and the effectiveness of the transformation parameter calculation is ensured;

[0122] According to the scene feature point coordinates and the virtual feature point coordinates in the same coordinate system, the spatial transformation parameters are calculated, including translation parameters, rotation parameters and scaling parameters: the translation parameters are used to represent the position offset of the virtual object relative to the real scene, the rotation parameters are used to describe the spatial rotation angle of the virtual object that needs to be adjusted, and the scaling parameters are used to match the size ratio of the virtual object and the real scene; the calculated translation parameters, rotation parameters and scaling parameters are integrated according to the matrix construction rule to generate an initial alignment transformation matrix, which completely contains all the spatial transformation information required for the alignment of the virtual object and the real scene;

[0123] S4. Adjust the preset attitude parameters of the virtual object according to the initial alignment transformation matrix, and correct the matrix parameters by combining the time sequence smoothing optimization algorithm to complete the real-time alignment of the real scene and the virtual object:

[0124] S4.1. Adjust the preset attitude parameters of the virtual object according to the initial alignment transformation matrix:

[0125] The translation parameters, rotation parameters and scaling parameters in the initial alignment transformation matrix are respectively applied to the preset posture parameter adjustment of the virtual object: the translation parameters directly act on the initial position parameters of the virtual object, adjust the spatial position of the virtual object in the AR scene according to the translation vector, and preliminarily move the virtual object to the position corresponding to the real scene; the rotation parameters adjust the initial rotation angle parameters of the virtual object according to the transformation rule of the rotation matrix, correct the spatial orientation of the virtual object, so that the posture of the virtual object is consistent with the viewing angle of the real scene; the scaling parameters are used to adjust the initial scaling ratio parameters of the virtual object, enlarge or reduce the size of the virtual object, so that the virtual object is coordinated with the size ratio of the objects in the real scene; through the coordinated adjustment of the three types of parameters, the position, angle and size of the virtual object in the three-dimensional space are updated, so that the virtual object is preliminarily adapted to the spatial environment of the real scene;

[0126] S4.2. Calculate the geometric error between the virtual object and the real scene to quantify the alignment degree:

[0127] Extract the edge contour features of the adjusted virtual object, and the edge contour features are obtained by extracting the surface edge point coordinates of the three-dimensional model of the virtual object and connecting to form a complete edge contour. At the same time, the edge contour features of the corresponding region of the real scene image are extracted, and the edge contour of the region corresponding to the position of the virtual object in the real scene is identified through an image edge detection algorithm to form a scene edge contour.

[0128] Calculate the geometric error between the two types of edge contour features, including contour overlap degree, distance deviation and other dimensions: the contour overlap degree is represented by calculating the overlap area ratio of the virtual object edge contour and the real scene edge contour, and the higher the overlap area ratio, the better the alignment effect; the distance deviation is quantified by calculating the average distance and maximum distance between the corresponding points on the two contours, and the smaller the distance value, the more accurate the contour alignment; through the comprehensive calculation of multi-dimensional geometric error, the alignment degree of the virtual object and the real scene is quantified comprehensively, and the error basis is provided for subsequent matrix correction;

[0129] S4.3. Correct the initial alignment transformation matrix through a time sequence smoothing optimization algorithm:

[0130] The time sequence smoothing optimization algorithm first extracts the change trend of the scene key feature points output by the lightweight convolutional neural network in continuous multiple frames of AR images, selects multiple adjacent frames of feature point data as analysis samples, establishes a time sequence correlation equation of the position change of the feature points, determines the time sequence constraint condition, and constructs a time sequence constraint model; the time sequence constraint model is used to limit the motion change amplitude of the virtual object in continuous frames, and ensure the smoothness of the motion trajectory;

[0131] Based on the geometric error calculated from the current frame, the initial alignment transformation matrix is iteratively adjusted: the objective is to minimize the comprehensive error, which includes the geometric error of the current frame and the time sequence constraint error calculated by comparing the feature point change amount of the current frame and the historical frame; the iteration step size is dynamically adjusted, with a larger step size in the early iteration to quickly approach the optimal solution, and the step size gradually decreases to improve the adjustment accuracy;

[0132] During each iteration process, the transformation matrix parameters of the current frame are corrected according to the change trend of the comprehensive error, including the fine tuning of translation parameters, rotation parameters and scaling parameters; the iteration convergence condition is judged by the stability of the CNN features, and the scene key feature point change amount obtained by continuous two iterations is continuously monitored. When the feature change amount is less than the preset range, it indicates that the matrix parameters have tended to be stable, the iteration is stopped and the corrected transformation matrix is output;

[0133] In some embodiments, the iteration convergence condition can be judged by the change amount of the comprehensive error, and the iteration is stopped when the difference between the comprehensive errors of continuous two iterations is less than a preset threshold; in specific implementation, the comprehensive error value of the last iteration is recorded first, the difference between the comprehensive error of the current iteration and the error of the last iteration is calculated, and when the difference remains below the preset threshold for multiple times, it is determined that the convergence condition is met, the iteration is stopped and the corrected transformation matrix is output. The two convergence judgment methods can be flexibly selected according to the error distribution and feature stability, ensuring that the optimization process is always based on the CNN feature output, and both can realize accurate correction of the matrix parameters, avoiding waste of computing resources caused by excessive iteration;

[0134] S4.4. Incorporate spatiotemporal consistency constraints and output the aligned AR image:

[0135] The corrected transformation matrix is applied to the pose adjustment of the virtual object again to ensure that the position, angle and size of the virtual object are fully adapted to the real scene; spatiotemporal consistency constraints are incorporated, which are based on physical motion laws to limit the motion trajectory of the virtual object in consecutive frames, avoiding sudden displacement, jumping rotation and other motion phenomena that do not conform to the physical laws;

[0136] The final verification is performed on the corrected virtual object pose, including whether the geometric error meets the preset requirements, whether the motion trajectory of the virtual object is smooth, and whether the fusion of the virtual object and the real scene is natural; after the verification is passed, the adjusted virtual object is superimposed and synthesized with the real scene image to generate an aligned AR image;

[0137] S4.5. Quality verification is performed on the aligned AR image to form a closed-loop optimization link:

[0138] After outputting the aligned AR image, quality verification is performed on the AR image, and the verification process is completed by a feature consistency detection branch of the lightweight CNN: the feature consistency detection branch includes a feature extraction sublayer and a consistency judgment sublayer, and the two layers are connected by a full connection manner; the feature extraction sublayer is used to extract the fusion features of the virtual object features and the real scene features in the aligned AR image, and the consistency judgment sublayer is used to analyze the consistency degree of the fusion features;

[0139] The training process of the feature consistency detection branch is as follows: a large number of AR image samples with fusion consistency and fusion inconsistency are collected as training data, and the samples are labeled and divided into a training set and a verification set after labeling; the training set samples are input into the branch network for training to learn the discrimination rule of feature consistency, and the difference between the predicted result and the label is calculated by using a cross-entropy loss function during the training process, and the branch network parameters are updated by using a gradient descent optimizer until the discrimination accuracy of the verification set is stable; the key parameters of the branch network include the convolution kernel size of the feature extraction sublayer, the feature dimension of the consistency judgment sublayer, and the activation parameters of the discrimination function;

[0140] The fusion consistency of the virtual object features and the real scene features is judged by the feature consistency detection branch, and when the consistency meets the preset requirement, the AR image is directly output; when the consistency does not meet the requirement, it is fed back to step S2 to adjust the feature extraction parameters of the lightweight convolutional neural network, including the convolution kernel weight of the multi-level feature extraction layer, the channel expansion coefficient of the enhanced Ghost module, and the weight distribution ratio of the feature fusion layer; After the subsequent alignment processes such as feature extraction, feature matching, matrix calculation and pose adjustment, a closed-loop optimization link with CNN as the core is formed to ensure that the finally output AR image has high-quality alignment effect;

[0141] In some embodiments, the key parameters of network training can also be adjusted according to the feedback results in the closed-loop optimization process, including the adjustment strategy of learning rate and the parameter setting of batch normalization, and the feature extraction capability of the network is further optimized by retraining;

[0142] Specifically, when the feedback shows that the feature extraction accuracy is insufficient, the decay rate of the learning rate can be appropriately reduced, and the training period of the network is prolonged, so that the network can learn the scene feature rules more fully; when the feedback shows that there are redundant features, the channel expansion coefficient of the enhanced Ghost module can be adjusted to reduce the generation of redundant features, so that the feature extraction is more targeted; this adjustment process does not need to reconstruct the network structure, and only by parameter optimization can the network performance be improved, ensuring the efficiency and feasibility of the closed-loop optimization;

[0143] Through the systematic step design and the cooperative work of each module, the embodiment realizes the real-time and accurate alignment of the real scene and the virtual object in the AR scene; through constructing the lightweight convolutional neural network containing the enhanced Ghost bottleneck structure, reasonably designing the hierarchical structure and connection relationship of the network, optimizing the training steps and key parameters, while controlling the model parameter quantity and the amount of calculation, the integrity and accuracy of the scene feature extraction are effectively improved, which provides a reliable feature basis for the alignment process; through the auxiliary correction of the feature matching by fusing the IMU inertial measurement data, the motion compensation model is constructed to correct the feature offset caused by the device motion, which reduces the influence of external factors on the alignment accuracy and improves the accuracy of the feature correspondence; through the combination of the time sequence smoothing optimization algorithm and the space-time consistency constraint, the time sequence constraint model is constructed and the iterative parameters are dynamically adjusted, which ensures the smoothness of the virtual object motion trajectory in the continuous frame and avoids the mutation phenomenon in the alignment process; the design of the closed-loop optimization link can timely find and correct the deviation in the feature extraction and alignment process, and through the parameter adjustment and process re-entry, the stability and reliability of the alignment effect are further improved, so that the fusion of the virtual object and the real scene in the aligned AR image is more natural; the overall method takes the CNN feature extraction as the core, supplemented by multi-data fusion and multi-algorithm optimization, which significantly improves the accuracy and stability of the AR scene image alignment under the premise of ensuring real-time, and provides reliable technical support for the landing application of AR related technologies.

[0144] The above only describes the preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, should not be regarded as excluding other embodiments, and can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the claims of the present application.

Claims

1. A method for real-time alignment of AR scenes with lightweight CNNs, characterized in that, The method comprises the following steps: S1. acquiring multi-view real scene images of an AR scene, three-dimensional model data of a virtual object, and IMU inertial measurement data; S2. constructing a lightweight convolutional neural network containing an enhanced Ghost bottleneck structure, extracting low-level structural features and high-level semantic features of the real scene images through the network, generating scene key feature points and feature reliability maps after feature pyramid fusion; S3. taking the scene key feature points as the core, assisting in fusing the IMU inertial measurement data and the three-dimensional model data of the virtual object, correcting the feature matching deviation through geometric correction, and generating an initial alignment transformation matrix; S4. adjusting the posture of the virtual object based on the initial alignment transformation matrix, correcting the matrix parameters by combining a time sequence smoothing optimization algorithm, and completing real-time alignment of the real scene and the virtual object; wherein step S2 comprises: S2.

1. building a lightweight convolutional neural network containing an input preprocessing layer, a multi-level feature extraction layer, an enhanced Ghost module, a feature fusion layer, and a parallel output branch, establishing a data transmission link between each layer through full connection, wherein the input preprocessing layer is used for image standardization processing, the multi-level feature extraction layer is used for hierarchical extraction of features of different dimensions, the enhanced Ghost module is used for feature expansion and optimization, the feature fusion layer is used for multi-scale feature integration, and the parallel output branch is used for generating target results respectively; S2.

2. performing pixel value normalization processing on the real scene image through the input preprocessing layer, mapping the pixel value to a preset range, adjusting the image size according to the input requirements of the lightweight convolutional neural network, ensuring that the image size matches the network input channel, inputting the processed image into the multi-level feature extraction layer, and obtaining edge and texture low-level structural features of different resolutions through convolution operation of different convolution kernel sizes; S2.

3. concatenating the low-level structural features output by the multi-level feature extraction layer in the channel dimension and inputting them into the enhanced Ghost module, generating rich feature representations containing more detailed information through feature conversion and expansion operations inside the module; S2.

4. layer-by-layer fusing features of different scales through the fusion layer of the feature pyramid structure, first performing up-sampling processing on the high-level features to make them consistent with the resolution of the low-level features, then performing feature superposition fusion, inputting the fused features into the parallel output branch, and generating scene key feature points and feature reliability maps representing feature matching confidence through convolution and activation operations in the branch respectively. In step S2.3, the enhanced Ghost module first reorganizes the low-level structural features through 1x1 convolution and aggregates the features to generate high-dimensional intrinsic feature maps; then, the intrinsic feature maps are expanded and the redundancy is suppressed through the combination of depth separable convolution and attention gate unit to generate complementary ghost feature maps; after the intrinsic feature maps and the ghost feature maps are element-wise weighted and fused, the rich feature representation after adaptation is output through batch normalization and activation function processing to adapt to subsequent fusion. The process is embedded in the feature extraction link of the lightweight convolutional neural network throughout, strengthening the network's ability to capture scene detail features.

2. The method of claim 1, wherein, Step S1 includes: S1.

1. Around the core interaction area of the AR scene, multi-view real scene images are collected at different orientations and different shooting angles. The shooting device parameters are kept consistent during the collection process, and the accurate time stamps of each image collection time are recorded. At the same time, the collected images are preliminarily filtered for noise; S1.

2. Obtain the complete three-dimensional model data of the virtual object, which includes vertex coordinate data, face index data, and texture mapping data. At the same time, obtain the preset attitude parameters of the virtual object, which include initial position parameters, initial rotation angle parameters, and initial scaling ratio parameters; S1.

3. Real-time collection of inertial measurement data of the IMU sensor built-in the AR device, including three-axis acceleration data, three-axis angular velocity data, and three-axis attitude angle data. Time stamp alignment processing is performed on the collected inertial measurement data to ensure correspondence with the collection time of the real scene images.

3. The method of claim 1, wherein, Step S3 includes: S3.

1. Read the feature reliability map output by the lightweight convolutional neural network, extract the confidence values of each pixel point in the feature reliability map, set a confidence threshold, and select scene key feature points with confidence values higher than the threshold to form a high-confidence feature point set; S3.

2. Extract surface feature points from the three-dimensional model data of the virtual object to establish a virtual feature point library. Use a feature distance matching algorithm to associate and match the feature points in the high-confidence feature point set with the feature points in the virtual feature point library, calculate the similarity of each pair of feature points, and determine the initial feature correspondence relationship according to the similarity sorting result; S3.

3. Auxiliary integration of preprocessed IMU inertial measurement data, based on the IMU data to calculate the motion trajectory and attitude change of the AR device, construct a motion offset model, and geometrically correct the feature point positions in the initial feature correspondence relationship according to the model to correct the feature offset caused by device motion; S3.

4. Based on the corrected feature correspondence relationship, extract the coordinate information of the corresponding feature points in the real scene image coordinate system and the virtual object three-dimensional coordinate system, calculate the spatial transformation parameters through the coordinate conversion formula, and integrate these parameters to generate an initial alignment transformation matrix.

4. The method of claim 1, wherein, Step S4 includes: S4.

1. Apply the translation, rotation and scaling parameters in the initial alignment transformation matrix to the preset posture parameters of the virtual object, and update the position, angle and size of the virtual object in three-dimensional space according to the matrix transformation rules, so that the virtual object can initially adapt to the spatial environment of the real scene. S4.

2. Extract the edge contour features of the adjusted virtual object and the edge contour features of the corresponding region of the real scene image, calculate the overlap and distance deviation geometric error between the two types of edge contour features, and quantify the alignment degree between the virtual object and the real scene. S4.

3. The initial alignment transformation matrix is ​​corrected by using a temporal smoothing optimization algorithm. First, the historical alignment parameters of multiple consecutive frames are introduced as a reference. The difference between the alignment parameters of the current frame and the historical frames is calculated. An optimization objective function is constructed by combining the geometric error. The minimum value of the optimization objective function is obtained by iteratively solving the problem. S4.

4. Incorporate spatiotemporal consistency constraints to ensure that the motion trajectory of virtual objects in consecutive frames conforms to the laws of physical motion, avoiding abrupt displacements. Perform a final verification on the corrected virtual object posture, and output the aligned AR image after the verification passes.

5. The method of claim 1, wherein, In step S2.4, the feature pyramid fusion layer adopts a bidirectional fusion strategy that combines top-down and bottom-up approaches. First, the high-level semantic features are upsampled and aligned with the low-level structural features, and then fused channel by channel. Next, the fused features are downsampled and adjusted to supplement detailed feature information. During the fusion process, the channel attention branch of the lightweight convolutional neural network dynamically allocates the weights of features at each scale, so that the fused features are more in line with the extraction requirements of key feature points in the scene.

6. The method of claim 3, wherein, In step S3.3, the IMU inertial measurement data is first calibrated with timestamps and corrected for zero bias. An initial estimate of the camera motion is obtained through pre-integration. Then, a motion compensation model is constructed based on the initial estimate to compensate the position of key feature points of the scene output by the lightweight convolutional neural network. By calculating the matching error between the compensated feature points and the features of the 3D model of the virtual object, the compensation parameters are iteratively corrected to finally obtain the corrected feature correspondence, thereby realizing the auxiliary correction role of IMU data for CNN feature matching.

7. The method of claim 4, wherein, In step S4.3, the temporal smoothing optimization algorithm first extracts the changing trend of key scene feature points output by the lightweight convolutional neural network in multiple consecutive frames and constructs a temporal constraint model; then, combined with the geometric error of the current frame, iteratively adjusts the parameters of the initial alignment transformation matrix; during the iteration process, the stability of the CNN features is used to determine the convergence condition of the iteration, and the iteration stops when the feature change is less than the preset range, ensuring that the optimization process is always based on the CNN feature output.

8. The method of claim 1, wherein, After step S4, the quality of the aligned AR image is verified by using the feature consistency detection branch of a lightweight CNN to determine the fusion consistency between virtual object features and real scene features. When the consistency requirement is not met, feedback is sent to step S2 to adjust the feature extraction parameters of the lightweight convolutional neural network, and the feature extraction and subsequent alignment process are repeated to form a closed-loop optimization link with CNN as the core.

Images