A device pose self-correction method, system and platform based on visual and pose feature collaborative fusion and a storage medium

Abstract

The application discloses a device pose self-correction method, system, platform and storage medium based on visual and pose feature collaborative fusion, and the scheme realizes dynamic adaptive deep fusion between visual information and sensor pose data by introducing a bidirectional cross attention mechanism, pose coding with an uncertainty parameter, and a joint optimization strategy combining geodesic line rotation loss and physical consistency loss. Moreover, the pose error caused by environmental occlusion or signal interference can be intelligently evaluated and corrected, while maintaining high-precision rotation estimation, and through image reconstruction auxiliary tasks and online timing optimization, the robustness, accuracy and physical reasonableness of mobile device pose estimation in complex outdoor scenes are improved, providing a reliable positioning basis for applications such as automatic driving and high-precision map collection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and spatial positioning technology, specifically relating to a device pose self-correction method, system, platform, and storage medium based on the collaborative fusion of visual and pose features. Background Technology

[0002] In the navigation and operation of outdoor mobile robots, accurately acquiring their own three-dimensional spatial pose is fundamental for path planning, environmental perception, and task operation. Currently, mainstream positioning solutions rely on the fusion of Global Navigation Satellite Systems (GNSS) and Inertial Measurement Units (IMUs). However, in scenarios such as urban canyons, tunnels, and tree-lined roads, satellite signals are easily affected by obstruction and multipath effects, leading to drift or even failure of positioning data; and the errors of IMUs accumulate over time, making it impossible to provide long-term stable absolute pose on their own.

[0003] To overcome these shortcomings, visual positioning technology has been widely adopted. However, in practical applications such as road inspection, purely visual methods face serious challenges: on the one hand, road surface textures are repetitive and features are sparse, making traditional feature point matching methods less robust; on the other hand, there are many dynamic interferences in the environment, such as vehicles, pedestrians, and tree shadows, which can contaminate visual information and lead to positioning failure.

[0004] Existing visual-inertial fusion schemes mostly employ filtering or optimization frameworks, treating visual observations and IMU data as two independent types of observations with loose or tight coupling. While these methods have some effectiveness, they typically assume a fixed sensor noise model, making them unable to dynamically adapt to changes in sensor reliability under different scenarios (such as fluctuating GPS signal strength). Furthermore, they lack proactive modeling and robust handling mechanisms for occlusion interference in visual information, and fail to fully utilize the powerful feature extraction and nonlinear fitting capabilities of deep learning for end-to-end intelligent fusion.

[0005] Therefore, in order to address the technical problems and deficiencies mentioned above, there is an urgent need to design and develop a device pose self-correction method, system, platform, and storage medium based on the collaborative fusion of visual and pose features. Summary of the Invention

[0006] To overcome the shortcomings and difficulties of the existing technology, the present invention aims to provide a device pose self-correction method, system, platform and storage medium based on the collaborative fusion of visual and pose features, so as to intelligently evaluate the reliability of sensors, actively combat visual interference, and achieve high-precision self-correction through deep neural network to achieve deep collaborative fusion of visual information and pose information.

[0007] The first objective of this invention is to provide a device pose self-correction method based on the collaborative fusion of visual and pose features; the second objective of this invention is to provide a device pose self-correction system based on the collaborative fusion of visual and pose features; the third objective of this invention is to provide a device pose self-correction platform based on the collaborative fusion of visual and pose features; and the fourth objective of this invention is to provide a computer-readable storage medium.

[0008] The first objective of this invention is achieved as follows: the method comprises:

[0009] First data and second data corresponding to the target scene are constructed and generated respectively; wherein, the first data is the visual image data of the target scene; and the second data is the initial pose data corresponding to the first data.

[0010] Create and generate third data corresponding to the first data, and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data;

[0011] Based on the cross-attention mechanism, the third data is used as the query feature, and the fourth data is used as the key feature and value feature for fusion processing to generate the corresponding fifth data; wherein, the fifth data is fused feature data;

[0012] Based on the fifth data, a sixth data corresponding to the target scene is created; wherein, the sixth data is the corrected target pose data.

[0013] Furthermore, before creating and generating the third data corresponding to the first data and the fourth data corresponding to the second data, the process further includes:

[0014] The first data is enhanced by random occlusion, and a seventh data corresponding to the first data is constructed and generated; wherein, the seventh data is the enhanced visual image;

[0015] The third data refers to the visual features of the enhanced visual image.

[0016] Furthermore, the step of creating and generating sixth data corresponding to the target scene based on the fifth data also includes:

[0017] Based on the fifth data, eighth data corresponding to the target scene is reconstructed and generated; wherein, the eighth data is the reconstructed visual image data of the target scene;

[0018] Based on the fifth data, the corresponding sixth and eighth data are generated synchronously through a parallel decoding branch.

[0019] Furthermore, the method further includes the step of:

[0020] The neural network model is trained using a joint loss function; wherein the joint loss function includes at least a geodesic rotation loss function for constraining the rotation component in the target pose data, and an image reconstruction loss function for constraining the difference between the eighth data and the first data.

[0021] The geodesic rotation loss function is constructed based on the geodesic distance between the predicted rotation quaternion and the actual rotation quaternion in the Lie algebra space.

[0022] Furthermore, the step of fusing the third data as a query feature and the fourth data as both key and value features based on the cross-attention mechanism to generate the corresponding fifth data also includes:

[0023] A mapping relationship between feature data and feature dimensions is constructed. Based on the mapping relationship, the third data and the fourth data are mapped to the same feature dimension, respectively. A ninth data corresponding to the third data and a tenth data corresponding to the fourth data are generated respectively. The ninth data is the mapped visual feature data, and the tenth data is the mapped pose feature data.

[0024] Using the ninth data as the query vector and the tenth data as the key vector and value vector respectively, the corresponding eleventh data is calculated and generated; wherein, the eleventh data is the relevance weight between the query vector and the key vector;

[0025] Based on the eleventh data, the value vector is weighted and aggregated to generate the corresponding fifth data.

[0026] Furthermore, the pose data includes a three-dimensional translation vector and a rotation quaternion;

[0027] After generating the sixth data corresponding to the target scene based on the fifth data, the process further includes: normalizing the rotation quaternions in the target pose data to ensure its unit modulus constraint.

[0028] The second objective of this invention is achieved as follows: the system is used to implement the device pose self-correction method based on the collaborative fusion of visual and pose features, the system comprising:

[0029] The first data construction and generation unit is used to construct and generate first data and second data corresponding to the target scene, respectively; wherein, the first data is visual image data of the target scene; and the second data is initial pose data corresponding to the first data.

[0030] The first data creation and generation unit is used to create and generate third data corresponding to the first data and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data.

[0031] The data fusion processing unit is used to fuse the third data as a query feature and the fourth data as a key feature and value feature based on the cross-attention mechanism, and generate corresponding fifth data; wherein the fifth data is fused feature data;

[0032] The second data creation and generation unit is used to create and generate a sixth data corresponding to the target scene based on the fifth data; wherein the sixth data is the corrected target pose data.

[0033] Furthermore, the system further includes: a second data construction and generation unit, used for random occlusion enhancement processing of the first data, and constructing and generating seventh data corresponding to the first data; wherein the seventh data is an enhanced visual image;

[0034] The third data is the visual characteristics of the enhanced visual image;

[0035] The first data processing unit is used to train the neural network model using a joint loss function; wherein the joint loss function includes at least a geodesic rotation loss function for constraining the rotation component in the target pose data, and an image reconstruction loss function for constraining the difference between the eighth data and the first data.

[0036] The second data processing unit is used to construct the geodesic rotation loss function based on the geodesic distance between the predicted rotation quaternion and the actual rotation quaternion in the Lie algebra space.

[0037] The data fusion processing unit further includes:

[0038] The first construction module is used to construct a mapping relationship between feature data and feature dimensions. Based on the mapping relationship, the third data and the fourth data are mapped to the same feature dimension respectively. The module also generates a ninth data corresponding to the third data and a tenth data corresponding to the fourth data respectively. The ninth data is the mapped visual feature data, and the tenth data is the mapped pose feature data.

[0039] The first generation module is used to calculate and generate the corresponding eleventh data by using the ninth data as the query vector and the tenth data as the key vector and value vector respectively; wherein, the eleventh data is the relevance weight between the query vector and the key vector;

[0040] The first processing module is used to perform weighted aggregation processing on the value vector based on the eleventh data, and to construct and generate the corresponding fifth data.

[0041] The second data creation and generation unit further includes:

[0042] The second construction module is used to reconstruct and generate eighth data corresponding to the target scene based on the fifth data; wherein the eighth data is the reconstructed visual image data of the target scene;

[0043] The second generation module is used to generate the corresponding sixth and eighth data synchronously based on the fifth data and through a parallel decoding branch;

[0044] The second processing module is used to normalize the rotation quaternions in the target pose data.

[0045] The pose data includes three-dimensional translation vectors and rotation quaternions to ensure unit modulus constraints.

[0046] The third objective of this invention is achieved as follows: it includes a processor, a memory, and a device pose self-correction platform control program based on the collaborative fusion of visual and pose features; wherein the device pose self-correction platform control program based on the collaborative fusion of visual and pose features is executed on the processor, the device pose self-correction platform control program based on the collaborative fusion of visual and pose features is stored in the memory, and the device pose self-correction platform control program based on the collaborative fusion of visual and pose features implements the device pose self-correction method based on the collaborative fusion of visual and pose features.

[0047] The fourth objective of this invention is achieved as follows: the computer-readable storage medium stores a device pose self-correction platform control program based on the collaborative fusion of visual and pose features, and the device pose self-correction platform control program based on the collaborative fusion of visual and pose features implements the device pose self-correction method based on the collaborative fusion of visual and pose features.

[0048] This invention constructs and generates first data and second data corresponding to a target scene through a method. The first data is visual image data of the target scene; the second data is initial pose data corresponding to the first data. A third data corresponding to the first data and a fourth data corresponding to the second data are created and generated. The third data is visual features of the visual image; the fourth data is pose features of the initial pose data. Based on a cross-attention mechanism, the third data is used as a query feature, and the fourth data is used as a key feature and a value feature for fusion processing to generate corresponding fifth data. The fifth data is fused feature data. Based on the fifth data, a sixth data corresponding to the target scene is created and generated. The sixth data is the corrected target pose data. The invention also includes a system, platform, and storage medium corresponding to the method. Through a deep neural network, dynamic and adaptive fusion of visual information and noisy initial pose information is achieved, effectively overcoming the influence of environmental occlusion and unstable sensor signals, thereby outputting a stable and accurate corrected pose.

[0049] In other words, the present invention achieves dynamic adaptive deep fusion between visual information and sensor pose data by introducing a bidirectional cross-attention mechanism, pose encoding with uncertain parameters, and a joint optimization strategy combining geodesic rotation loss and physical consistency loss. Furthermore, it can intelligently assess and correct pose errors caused by environmental occlusion or signal interference. While maintaining high-precision rotation estimation, it improves the robustness, accuracy, and physical plausibility of mobile device pose estimation in complex outdoor scenarios through image reconstruction assistance and online temporal optimization, providing a reliable positioning foundation for applications such as autonomous driving and high-precision map acquisition. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the process steps of a device pose self-correction method based on the collaborative fusion of visual and pose features according to the present invention.

[0052] Figure 2 This is a schematic diagram of the overall architecture of a device pose self-correction method based on the collaborative fusion of visual and pose features according to the present invention;

[0053] Figure 3This is a schematic diagram of the overall architecture of the pose correction neural network model constructed by the present invention, which is a device pose self-correction method based on the collaborative fusion of visual and pose features.

[0054] Figure 4 This is a detailed processing flowchart of the vision-position collaborative feature fusion module of the device pose self-correction method based on the collaborative fusion of vision and pose features of the present invention.

[0055] Figure 5 This is a schematic diagram of a device pose self-correction system architecture based on the collaborative fusion of visual and pose features according to the present invention;

[0056] Figure 6 This is a schematic diagram of a device pose self-correction platform architecture based on the collaborative fusion of visual and pose features according to the present invention;

[0057] Figure 7 This is a schematic diagram of a computer-readable storage medium architecture in one embodiment of the present invention. Detailed Implementation

[0058] To facilitate a clearer understanding of the objectives, technical solutions, and advantages of this invention, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Those skilled in the art can easily understand other advantages and effects of this invention from the content disclosed in this specification.

[0059] This invention can also be implemented or applied through other different specific examples, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of this invention.

[0060] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0061] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Secondly, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0062] Preferably, the device pose self-correction method based on the collaborative fusion of visual and pose features of the present invention is applied in one or more terminals or servers. The terminal is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0063] The terminal can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal can interact with the customer via a keyboard, mouse, remote control, touchpad, or voice control device.

[0064] This invention provides a device pose self-correction method, system, platform, and storage medium based on the collaborative fusion of visual and pose features.

[0065] like Figure 1 The diagram shown is a flowchart of a device pose self-correction method based on the collaborative fusion of visual and pose features provided in an embodiment of the present invention.

[0066] In this embodiment, the device pose self-correction method based on the collaborative fusion of visual and pose features can be applied to terminals with display functions or fixed terminals. The terminals are not limited to personal computers, smartphones, tablets, desktop computers or all-in-one computers with cameras, etc.

[0067] The device pose self-correction method based on the collaborative fusion of visual and pose features can also be applied to a hardware environment consisting of a terminal and a server connected to the terminal via a network. The network includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The device pose self-correction method based on the collaborative fusion of visual and pose features in this embodiment can be executed by the server, by the terminal, or by both the server and the terminal.

[0068] For example, for a device pose self-correction terminal that requires collaborative fusion of visual and pose features, the device pose self-correction function based on the collaborative fusion of visual and pose features provided by the method of this invention can be directly integrated into the terminal, or a client for implementing the method of this invention can be installed. Alternatively, the method provided by this invention can also run on servers or other devices in the form of a Software Development Kit (SDK), providing an interface for the device pose self-correction function based on the collaborative fusion of visual and pose features. Terminals or other devices can then implement the device pose self-correction function based on the collaborative fusion of visual and pose features through the provided interface. The invention will be further described below with reference to the accompanying drawings.

[0069] like Figures 1-4 As shown, this invention provides a device pose self-correction method based on the collaborative fusion of visual and pose features. The method includes the following steps:

[0070] S01. Construct and generate first data and second data corresponding to the target scene respectively; wherein, the first data is the visual image data of the target scene; and the second data is the initial pose data corresponding to the first data;

[0071] S02. Create and generate third data corresponding to the first data and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data;

[0072] S03. Based on the cross-attention mechanism, the third data is used as the query feature, and the fourth data is used as the key feature and value feature for fusion processing to generate the corresponding fifth data; wherein, the fifth data is fused feature data;

[0073] S04. Based on the fifth data, create and generate sixth data corresponding to the target scene; wherein, the sixth data is the corrected target pose data.

[0074] Before creating and generating the third data corresponding to the first data and the fourth data corresponding to the second data, the process further includes:

[0075] S0201. The first data is subjected to random occlusion enhancement processing, and a seventh data corresponding to the first data is constructed and generated; wherein, the seventh data is the enhanced visual image;

[0076] The third data refers to the visual features of the enhanced visual image.

[0077] The step of creating and generating sixth data corresponding to the target scene based on the fifth data further includes:

[0078] S041. Based on the fifth data, reconstruct and generate eighth data corresponding to the target scene; wherein, the eighth data is the reconstructed visual image data of the target scene;

[0079] S042. Based on the fifth data, the corresponding sixth and eighth data are generated synchronously through parallel decoding branches.

[0080] The method further includes the following steps:

[0081] S051. The neural network model is trained using a joint loss function; wherein the joint loss function includes at least a geodesic rotation loss function for constraining the rotation component in the target pose data, and an image reconstruction loss function for constraining the difference between the eighth data and the first data.

[0082] The geodesic rotation loss function is constructed based on the geodesic distance between the predicted rotation quaternion and the actual rotation quaternion in the Lie algebra space.

[0083] The method based on cross-attention, using the third data as a query feature, fuses the fourth data as both key and value features to generate corresponding fifth data, and further includes:

[0084] S031. Construct a mapping relationship between feature data and feature dimensions. Based on the mapping relationship, map the third data and the fourth data to the same feature dimension respectively. And generate a ninth data corresponding to the third data and a tenth data corresponding to the fourth data respectively. The ninth data is the mapped visual feature data. The tenth data is the mapped pose feature data.

[0085] S032. Using the ninth data as the query vector and the tenth data as the key vector and value vector respectively, calculate and generate the corresponding eleventh data; wherein, the eleventh data is the relevance weight between the query vector and the key vector;

[0086] S033. Based on the eleventh data, the value vector is weighted and aggregated to generate the corresponding fifth data.

[0087] The pose data includes three-dimensional translation vectors and rotation quaternions;

[0088] After creating and generating sixth data corresponding to the target scene based on the fifth data, the process further includes:

[0089] S043. Normalize the rotation quaternions in the target pose data to ensure their unit modulus constraint.

[0090] Specifically, in this embodiment of the invention, a high-precision self-calibration method for the pose of road inspection equipment with resistance to occlusion interference is provided, comprising the following steps:

[0091] Step S1: Data Acquisition and Preprocessing. Acquire road surface images collected by the inspection equipment and the corresponding initial sensor pose data; the initial pose data includes three-dimensional position coordinates and quaternion rotation parameters;

[0092] Step S2: Anti-interference enhancement. During the training phase, random positions and sizes of solid-color occlusion blocks are applied to the road surface image to simulate dynamic road occlusion and generate an enhanced input image;

[0093] Step S3: Two-stream feature extraction. The input image is fed into a deep convolutional neural network to extract visual feature vectors; simultaneously, the initial pose data is fed into a multilayer perceptron to extract geometric pose feature vectors.

[0094] Step S4: View-position collaborative feature fusion. Using a cross-attention mechanism, the visual feature vector is mapped to a query vector, and the geometric pose feature vector is mapped to a key vector and a value vector. Attention weights are calculated and multimodal fusion features are generated.

[0095] S41: Construct a global prior memory. Initialize a learnable global memory matrix in the network to store general geometric prior features of the road scene extracted from the training set;

[0096] S42: First-order pose guidance fusion. Using a cross-attention mechanism, the visual feature vector is mapped to a first query vector (Query 1), and the geometric pose feature vector is mapped to a first key vector (Key 1) and a first value vector (Value 1); the first attention weights are calculated and preliminary pose guidance features are generated;

[0097] S43: Second-order memory-enhanced retrospective fusion. The initial pose guidance features are used as the second query vector (Query 2), and the concatenation result of the global memory matrix and the original visual feature vector is used as the second key vector (Key 2) and the second value vector (Value 2).

[0098] S44: Multimodal output generation. Based on the interaction between the second query vector and the second key vector, the second attention weight containing prior constraints is calculated, and the second value vector is weighted and aggregated to generate the final multimodal fusion feature.

[0099] Step S5: Multi-task prediction. The multimodal fusion features are simultaneously input into the pose regression decoder and the image reconstruction decoder; the pose regression decoder outputs the corrected six-DOF pose; the image reconstruction decoder outputs the reconstructed road surface image;

[0100] S51: Semantic-assisted parsing. A lightweight semantic segmentation branch is established, and the multimodal fusion features are input into the semantic decoder to predict the pixel-level semantic category mask of the image. The categories include at least road surface region, sky region, and dynamic vehicle region.

[0101] S52: Semantic-guided feature filtering. A static weight map is generated using the predicted semantic category mask, and the multimodal fusion features are weighted and filtered to suppress feature responses in dynamic vehicle regions and enhance feature weights in road and sky regions;

[0102] S53: Pose Regression Prediction. The filtered features are input into the pose regression decoder, which outputs the corrected six-DOF pose.

[0103] S54: Self-supervised image reconstruction. The multimodal fusion features are input into the image reconstruction decoder, which outputs the reconstructed road surface image. A self-supervised learning signal is provided through pixel-level reconstruction error to constrain the feature extractor to maintain global contextual information.

[0104] Step S6: Joint Optimization Training. A total loss function is constructed based on the geodesic rotation loss and image pixel reconstruction loss, and the network parameters are iteratively optimized end-to-end.

[0105] S61: Physical consistency geodesic loss. Calculate the geodesic distance between the predicted quaternion and the true quaternion in the Lie algebra space, as the rotational physical consistency loss;

[0106] S62: Virtual viewpoint cyclic consistency check. Using the corrected pose, the original road image is spatially transformed through a differentiable transformation module to synthesize a virtual viewpoint image; this virtual viewpoint image is then input into a shared-weight pose regression network to obtain a secondary predicted pose; the difference between the secondary predicted pose and the theoretical transformed pose is calculated as the cyclic consistency loss;

[0107] The cross-attention mechanism in step S4 is as follows: the visual feature vector is projected to the same dimension as the geometric pose feature vector through a linear projection layer; the correlation matrix between the visual features and the pose features is calculated using the dot product attention formula; and the geometric pose features are weighted and aggregated according to the correlation matrix, so that the network dynamically corrects the pose features according to the saliency of the image texture.

[0108] The total loss function in step S6 The calculation formula is as follows:

[0109] (6)

[0110] In the formula, This represents the mean square error loss of the position coordinates; This is a geodesic rotation loss based on quaternion logarithmic mapping, used to measure the shortest path distance between the predicted rotation and the true rotation in the Lie algebra space; Huber loss or L2 loss for image reconstruction; These are the weighting coefficients for each part. , , These are the weighting coefficients for each part.

[0111] The attitude regression decoder includes a normalization layer at the output end, which is used to force the output quaternion vector to have a magnitude of 1, thus ensuring the legality of geometric rotation.

[0112] Example 1

[0113] Data Definition and Augmentation: In this embodiment, the pose vector of the road inspection device is defined as follows: .in, Represents a three-dimensional translation vector. Represent a rotation vector in quaternion form that satisfies the unit norm constraint. The input image is represented as To simulate occlusion interference, a mask matrix is ​​constructed. The occluded area is represented by 0, and the unoccluded area by 1. (Enhanced input image) The calculation formula is:

[0114] (1)

[0115] In the formula, This indicates an element-wise multiplication operation. This represents the road RGB image tensor of the original input; This represents a randomly generated binary occlusion mask matrix (where the occluded area has a value of 0 and the unoccluded area has a value of 1).

[0116] Example 2: Feature Extraction and Attention Fusion

[0117] The visual features extracted by the image encoder of the neural network are denoted as... The geometric features extracted by the attitude encoder are denoted as To align dimensions, a linear projection matrix is ​​used. Map features to a unified hidden layer dimension .

[0118] The calculation process for cross-attention used in this invention is as follows:

[0119] First, generate the query vector. Key vector Sum value vector :

[0120] (2)

[0121] In the formula, This represents the high-dimensional feature vector of the image extracted by the convolutional neural network. Let be the projection weight matrix representing the learnable image features.

[0122] (3)

[0123] In the formula, The weight matrix is ​​used to represent the projected weights of the learnable pose feature key vectors. The projection weight matrix represents the learnable pose feature vector.

[0124] Subsequently, the attention weight matrix is ​​calculated and fused features are generated. :

[0125] (4)

[0126] In the formula, The dimension of the feature space is used as a scaling factor to prevent the gradient from vanishing due to excessively large dot product values. This demonstrates the matrix transpose operation.

[0127] Using this formula, the network can determine visual features. The saliency of the input noise posture characteristics is dynamically adjusted. The level of attention paid to this allows for the filtering out of unreliable sensor noise.

[0128] Example 3: Output and Normalization

[0129] At the output of the pose regression decoder, the network outputs the predicted translation vector. and unnormalized rotation quaternions To ensure the geometric validity of the rotation, the quaternion is normalized using the L2 norm to obtain the final predicted quaternion. :

[0130] (5)

[0131] In the formula, This represents the unnormalized quaternion prediction vector directly output by the attitude regression decoder; The L2 norm (Euclidean norm) operation of a vector is represented.

[0132] The final output corrected pose is .

[0133] Example 4: Construction of Loss Function

[0134] This invention designs a composite loss function. The formula used to guide network training is as follows:

[0135] (6)

[0136] In the formula, This represents the mean square error loss of the position coordinates; This is a geodesic rotation loss based on quaternion logarithmic mapping, used to measure the shortest path distance between the predicted rotation and the true rotation in the Lie algebra space; Huber loss or L2 loss for image reconstruction; These are the weighting coefficients for each part. , , These are the weighting coefficients for each part.

[0137] Location loss The accuracy of the translation vector is measured using the mean square error (MSE).

[0138] (7)

[0139] In the formula, Labels for actual translation vectors; This represents the translation vector predicted by the network.

[0140] Geodesic rotation loss ( To accurately measure differences in a three-dimensional rotating space (SO(3) manifold), this invention does not directly use the Euclidean distance of quaternions, but instead calculates the predicted quaternions. With real quaternions The geodesic distance between them.

[0141] First, calculate the difference quaternion. :

[0142] (8)

[0143] In the formula, To represent quaternion multiplication, Represents the conjugate (inverse) of a real quaternion; This represents the predicted and normalized quaternion.

[0144] Subsequently, based on the real part of the quaternion Calculate the rotation angle error:

[0145] (9)

[0146] In the formula, Represents the real part (scalar part) of the difference quaternion $q_{diff}$; This represents the truncation function, used to ensure that the value is within the range of [0, 1], preventing the inverse cosine function from exceeding the limit.

[0147] This loss function directly reflects the minimum rotation angle difference between two poses and has a clear physical meaning.

[0148] Image reconstruction loss ( To enhance robustness to occlusion, the Huber loss function is used to constrain the image reconstruction. Consistency with the original unoccluded image:

[0149] (10)

[0150] In the formula, To represent the threshold parameter set for the Huber loss function; This represents the pixel value at coordinates (i,j) of the original unoccluded image. This represents the predicted pixel value of the reconstructed image at coordinates (i,j).

[0151] Example 5: Application Scenario Examples and Experimental Data Comparison and Verification

[0152] To verify the effectiveness of the present invention in complex outdoor scenarios, especially the robustness, accuracy and physical rationality of mobile device pose estimation, this embodiment selects typical difficult scenarios in actual road inspection for comparative experiments.

[0153] Experimental Environment and Dataset Construction. Experimental data was collected from a municipal road inspection project, with the inspection vehicles equipped with the device traveling at speeds ranging from 40 km / h to 60 km / h. To comprehensively evaluate the algorithm's performance, the following three challenging typical application scenarios were selected:

[0154] Scenario A (Severe Obstruction Scenario): A congested section of a main urban road, with a large bus or truck in front, obstructing more than 40% of the view, and there is frequent dynamic vehicle interference.

[0155] Scene B (Scene with drastic changes in lighting): When entering and exiting tunnel entrances and overpass shadow areas, the image brightness undergoes a rapid change with high dynamic range.

[0156] Scene C (weak texture scene): Newly paved asphalt road surface with a simple texture and lack of obvious traffic signs or curb features.

[0157] Comparative Scheme Setup. To demonstrate the superiority of the proposed solution, the following two sets of comparative examples and the proposed solution are tested side-by-side:

[0158] Comparative Example 1 (Traditional Geometric Method): The classic ORB-SLAM2 algorithm is used, which relies solely on image feature points for pose calculation.

[0159] Comparative Example 2 (Basic Deep Learning Method): The basic PoseNet network (ResNet backbone + direct regression) is used, with Euclidean distance loss (MSELoss) and no cross-attention mechanism or semantic auxiliary task is introduced.

[0160] The present invention adopts the complete method described in Examples 1 to 4, including cross-attention fusion, semantic anti-occlusion branching, geodesic rotation loss, and cyclic consistency constraint.

[0161] Comparison of experimental results. The main evaluation indicators are: mean absolute translation error (ATE, m) and mean absolute rotation error (ARE, degree). The smaller the error value, the higher the accuracy.

[0162] Table 1: Comparison of pose estimation accuracy in different scenarios

[0163]

[0164] To achieve the above objectives, the present invention also provides a device pose self-correction system based on the collaborative fusion of visual and pose features, such as... Figure 5 As shown, the system is applied to the device pose self-correction method based on the collaborative fusion of visual and pose features. The system includes:

[0165] The first data construction and generation unit is used to construct and generate first data and second data corresponding to the target scene, respectively; wherein, the first data is visual image data of the target scene; and the second data is initial pose data corresponding to the first data.

[0166] The first data creation and generation unit is used to create and generate third data corresponding to the first data and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data.

[0167] The data fusion processing unit is used to fuse the third data as a query feature and the fourth data as a key feature and value feature based on the cross-attention mechanism, and generate corresponding fifth data; wherein the fifth data is fused feature data;

[0168] The second data creation and generation unit is used to create and generate a sixth data corresponding to the target scene based on the fifth data; wherein the sixth data is the corrected target pose data.

[0169] The system further includes: a second data construction and generation unit, used for random occlusion enhancement processing of the first data, and constructing and generating seventh data corresponding to the first data; wherein, the seventh data is an enhanced visual image;

[0170] The third data is the visual characteristics of the enhanced visual image;

[0171] The first data processing unit is used to train the neural network model using a joint loss function; wherein the joint loss function includes at least a geodesic rotation loss function for constraining the rotation component in the target pose data, and an image reconstruction loss function for constraining the difference between the eighth data and the first data.

[0172] The second data processing unit is used to construct the geodesic rotation loss function based on the geodesic distance between the predicted rotation quaternion and the actual rotation quaternion in the Lie algebra space.

[0173] The data fusion processing unit further includes:

[0174] The first construction module is used to construct a mapping relationship between feature data and feature dimensions. Based on the mapping relationship, the third data and the fourth data are mapped to the same feature dimension respectively. The module also generates a ninth data corresponding to the third data and a tenth data corresponding to the fourth data respectively. The ninth data is the mapped visual feature data, and the tenth data is the mapped pose feature data.

[0175] The first generation module is used to calculate and generate the corresponding eleventh data by using the ninth data as the query vector and the tenth data as the key vector and value vector respectively; wherein, the eleventh data is the relevance weight between the query vector and the key vector;

[0176] The first processing module is used to perform weighted aggregation processing on the value vector based on the eleventh data, and to construct and generate the corresponding fifth data.

[0177] The second data creation and generation unit further includes:

[0178] The second construction module is used to reconstruct and generate eighth data corresponding to the target scene based on the fifth data; wherein the eighth data is the reconstructed visual image data of the target scene;

[0179] The second generation module is used to generate the corresponding sixth and eighth data synchronously based on the fifth data and through a parallel decoding branch;

[0180] The second processing module is used to normalize the rotation quaternions in the target pose data.

[0181] The pose data includes three-dimensional translation vectors and rotation quaternions to ensure unit modulus constraints.

[0182] In another embodiment of the present invention, to achieve the objective of the present invention, a high-precision self-correction system for the pose of a road inspection device with anti-obstruction interference is also provided, for performing the method described above. The system includes: a data acquisition module for acquiring road video streams and GPS / IMU coarse positioning data; a processor module with a built-in trained pose correction neural network model; and a storage module for storing road image data and corrected high-precision pose logs.

[0183] In the system solution embodiment of the present invention, the specific details of the method steps involved in the device pose self-correction based on the collaborative fusion of visual and pose features have been described above. That is to say, the functional modules in the system are used to implement the steps or sub-steps in the above method embodiment, which will not be repeated here.

[0184] To achieve the above objectives, the present invention also provides a device pose self-correction platform based on the collaborative fusion of visual and pose features, such as... Figure 6 As shown, the system includes a processor, a memory, and a device pose self-correction platform control program based on the collaborative fusion of visual and pose features. The processor executes the device pose self-correction platform control program based on the collaborative fusion of visual and pose features, and the program is stored in the memory. This control program implements the steps of the device pose self-correction method based on the collaborative fusion of visual and pose features, for example:

[0185] S01. Construct and generate first data and second data corresponding to the target scene respectively; wherein, the first data is the visual image data of the target scene; and the second data is the initial pose data corresponding to the first data;

[0186] S02. Create and generate third data corresponding to the first data and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data;

[0187] S03. Based on the cross-attention mechanism, the third data is used as the query feature, and the fourth data is used as the key feature and value feature for fusion processing to generate the corresponding fifth data; wherein, the fifth data is fused feature data;

[0188] S04. Based on the fifth data, create and generate sixth data corresponding to the target scene; wherein, the sixth data is the corrected target pose data.

[0189] The specific details of the steps have been explained above and will not be repeated here.

[0190] In this embodiment of the invention, the built-in processor of the device pose self-correction platform based on the collaborative fusion of vision and pose features can be composed of integrated circuits. For example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor connects to various components using various interfaces and lines, and executes programs or units stored in memory, and calls data stored in memory, to perform various functions of device pose self-correction based on the collaborative fusion of vision and pose features and process data.

[0191] The memory is used to store program code and various data. It is installed in the device pose self-correction platform based on the collaborative fusion of vision and pose features, and enables high-speed and automatic access to programs or data during operation. The memory includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0192] To achieve the above objectives, the present invention also provides a computer-readable storage medium, such as... Figure 7 As shown, the computer-readable storage medium stores a device pose self-correction platform control program based on the collaborative fusion of visual and pose features. This control program implements the steps of the device pose self-correction method based on the collaborative fusion of visual and pose features; for example:

[0193] S01. Construct and generate first data and second data corresponding to the target scene respectively; wherein, the first data is the visual image data of the target scene; and the second data is the initial pose data corresponding to the first data;

[0194] S02. Create and generate third data corresponding to the first data and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data;

[0195] S03. Based on the cross-attention mechanism, the third data is used as the query feature, and the fourth data is used as the key feature and value feature for fusion processing to generate the corresponding fifth data; wherein, the fifth data is fused feature data;

[0196] S04. Based on the fifth data, create and generate sixth data corresponding to the target scene; wherein, the sixth data is the corrected target pose data.

[0197] The specific details of the steps have been explained above and will not be repeated here.

[0198] In the description of embodiments of the present invention, it should be noted that any process or method description in the flowcharts or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0199] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, a “computer-readable medium” can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0200] This invention constructs and generates first data and second data corresponding to a target scene through a method. The first data is visual image data of the target scene; the second data is initial pose data corresponding to the first data. A third data corresponding to the first data and a fourth data corresponding to the second data are created and generated. The third data is visual features of the visual image; the fourth data is pose features of the initial pose data. Based on a cross-attention mechanism, the third data is used as a query feature, and the fourth data is used as a key feature and a value feature for fusion processing to generate corresponding fifth data. The fifth data is fused feature data. Based on the fifth data, a sixth data corresponding to the target scene is created and generated. The sixth data is the corrected target pose data. The invention also includes a system, platform, and storage medium corresponding to the method. Through a deep neural network, dynamic and adaptive fusion of visual information and noisy initial pose information is achieved, effectively overcoming the influence of environmental occlusion and unstable sensor signals, thereby outputting a stable and accurate corrected pose.

[0201] In other words, the present invention achieves dynamic adaptive deep fusion between visual information and sensor pose data by introducing a bidirectional cross-attention mechanism, pose encoding with uncertain parameters, and a joint optimization strategy combining geodesic rotation loss and physical consistency loss. Furthermore, it can intelligently assess and correct pose errors caused by environmental occlusion or signal interference. While maintaining high-precision rotation estimation, it improves the robustness, accuracy, and physical plausibility of mobile device pose estimation in complex outdoor scenarios through image reconstruction assistance and online temporal optimization, providing a reliable positioning foundation for applications such as autonomous driving and high-precision map acquisition.

[0202] In other words, the present invention acquires RGB images of the road and initial noise pose data captured by an onboard camera; performs random occlusion enhancement processing on the images to simulate road environment interference; extracts high-dimensional features of the images through a convolutional neural network and encodes pose features through a multilayer perceptron; utilizes a cross-attention mechanism to perform dynamic feature fusion with image features as query vectors and pose features as key-value vectors; inputs the fused features into the pose regression branch and the image reconstruction branch respectively, and outputs the corrected high-precision pose and reconstructed image. That is, by utilizing the cross-attention mechanism, the dynamic fusion of visual and geometric information is achieved, enabling the network to automatically adjust its trust in the initial pose data based on the saliency of the image content, thus improving the effectiveness of feature fusion; by introducing a geodesic rotation loss function, compared to the traditional Euclidean distance loss, it can more accurately describe the rotation error in three-dimensional space, improving the accuracy of pose correction; through random occlusion data enhancement and image reconstruction auxiliary tasks, the network is forced to learn global contextual information of the road surface, thereby enhancing the robustness of the algorithm in complex environments such as vehicle occlusion and tree shadow interference.

[0203] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A device pose self-correction method based on the collaborative fusion of visual and pose features, characterized in that, The method includes: First data and second data corresponding to the target scene are constructed and generated respectively; wherein, the first data is the visual image data of the target scene; and the second data is the initial pose data corresponding to the first data. Create and generate third data corresponding to the first data, and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data; Based on the cross-attention mechanism, the third data is used as the query feature, and the fourth data is used as the key feature and value feature for fusion processing to generate the corresponding fifth data; wherein, the fifth data is fused feature data; Based on the fifth data, a sixth data corresponding to the target scene is created; wherein, the sixth data is the corrected target pose data.

2. The device pose self-correction method based on the collaborative fusion of visual and pose features according to claim 1, characterized in that, Before creating and generating the third data corresponding to the first data and the fourth data corresponding to the second data, the process further includes: The first data is enhanced by random occlusion, and a seventh data corresponding to the first data is constructed and generated; wherein, the seventh data is the enhanced visual image; The third data refers to the visual features of the enhanced visual image.

3. A device pose self-correction method based on the collaborative fusion of visual and pose features according to claim 1 or 2, characterized in that, The step of creating and generating sixth data corresponding to the target scene based on the fifth data further includes: Based on the fifth data, eighth data corresponding to the target scene is reconstructed and generated; wherein, the eighth data is the reconstructed visual image data of the target scene; Based on the fifth data, the corresponding sixth and eighth data are generated synchronously through a parallel decoding branch.

4. The device pose self-correction method based on the collaborative fusion of visual and pose features according to claim 3, characterized in that, The method further includes the following steps: The neural network model is trained using a joint loss function; wherein the joint loss function includes at least a geodesic rotation loss function for constraining the rotation component in the target pose data, and an image reconstruction loss function for constraining the difference between the eighth data and the first data. The geodesic rotation loss function is constructed based on the geodesic distance between the predicted rotation quaternion and the actual rotation quaternion in the Lie algebra space.

5. The device pose self-correction method based on the collaborative fusion of visual and pose features according to claim 1, characterized in that, The method based on cross-attention, using the third data as a query feature, fuses the fourth data as both key and value features to generate corresponding fifth data, and further includes: A mapping relationship between feature data and feature dimensions is constructed. Based on the mapping relationship, the third data and the fourth data are mapped to the same feature dimension, respectively. A ninth data corresponding to the third data and a tenth data corresponding to the fourth data are generated respectively. The ninth data is the mapped visual feature data, and the tenth data is the mapped pose feature data. Using the ninth data as the query vector and the tenth data as the key vector and value vector respectively, the corresponding eleventh data is calculated and generated; wherein, the eleventh data is the relevance weight between the query vector and the key vector; Based on the eleventh data, the value vector is weighted and aggregated to generate the corresponding fifth data.

6. The device pose self-correction method based on the collaborative fusion of visual and pose features according to claim 3, characterized in that, The pose data includes three-dimensional translation vectors and rotation quaternions; After generating the sixth data corresponding to the target scene based on the fifth data, the process further includes: normalizing the rotation quaternions in the target pose data.

7. A device pose self-correction system based on the collaborative fusion of visual and pose features, characterized in that, The system is applied to the device pose self-correction method based on the collaborative fusion of visual and pose features as described in any one of claims 1 to 6, and the system comprises: The first data construction and generation unit is used to construct and generate first data and second data corresponding to the target scene, respectively; wherein, the first data is visual image data of the target scene; and the second data is initial pose data corresponding to the first data. The first data creation and generation unit is used to create and generate third data corresponding to the first data and fourth data corresponding to the second data; wherein, the third data is the visual features of the visual image; and the fourth data is the pose features of the initial pose data. The data fusion processing unit is used to fuse the third data as a query feature and the fourth data as a key feature and value feature based on the cross-attention mechanism, and generate corresponding fifth data; wherein the fifth data is fused feature data; The second data creation and generation unit is used to create and generate a sixth data corresponding to the target scene based on the fifth data; wherein the sixth data is the corrected target pose data.

8. A device pose self-correction system based on the collaborative fusion of visual and pose features according to claim 7, characterized in that, The system also includes: The second data construction and generation unit is used to perform random occlusion enhancement processing on the first data and construct and generate seventh data corresponding to the first data; wherein, the seventh data is an enhanced visual image; The third data is the visual characteristics of the enhanced visual image; The first data processing unit is used to train the neural network model using a joint loss function; wherein the joint loss function includes at least a geodesic rotation loss function for constraining the rotation component in the target pose data, and an image reconstruction loss function for constraining the difference between the eighth data and the first data. The second data processing unit is used to construct the geodesic rotation loss function based on the geodesic distance between the predicted rotation quaternion and the actual rotation quaternion in the Lie algebra space. The data fusion processing unit further includes: The first construction module is used to construct a mapping relationship between feature data and feature dimensions. Based on the mapping relationship, the third data and the fourth data are mapped to the same feature dimension respectively. The module also generates a ninth data corresponding to the third data and a tenth data corresponding to the fourth data respectively. The ninth data is the mapped visual feature data, and the tenth data is the mapped pose feature data. The first generation module is used to calculate and generate the corresponding eleventh data by using the ninth data as the query vector and the tenth data as the key vector and value vector respectively; wherein, the eleventh data is the relevance weight between the query vector and the key vector; The first processing module is used to perform weighted aggregation processing on the value vector based on the eleventh data, and to construct and generate the corresponding fifth data. The second data creation and generation unit further includes: The second construction module is used to reconstruct and generate eighth data corresponding to the target scene based on the fifth data; wherein the eighth data is the reconstructed visual image data of the target scene; The second generation module is used to generate the corresponding sixth and eighth data synchronously based on the fifth data and through a parallel decoding branch; The second processing module is used to normalize the rotation quaternions in the target pose data. The pose data includes three-dimensional translation vectors and rotation quaternions.

9. A device pose self-correction platform based on the collaborative fusion of visual and pose features, characterized in that, The system includes a processor, a memory, and a device pose self-correction platform control program based on the collaborative fusion of visual and pose features. The processor executes the device pose self-correction platform control program based on the collaborative fusion of visual and pose features, and the program is stored in the memory. The device pose self-correction platform control program based on the collaborative fusion of visual and pose features implements the device pose self-correction method based on the collaborative fusion of visual and pose features as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a device pose self-correction platform control program based on the collaborative fusion of visual and pose features. The device pose self-correction platform control program based on the collaborative fusion of visual and pose features implements the device pose self-correction method based on the collaborative fusion of visual and pose features as described in any one of claims 1 to 6.

Images