Semantic visual odometry-based pose estimation methods, devices, and media

Abstract

This invention discloses a pose estimation method, device, and medium based on semantic visual odometry. The method includes: inputting a semantic image sequence of a target object into a lightweight feature extraction network to extract continuous frame semantic features that describe spatial structure information; for each frame semantic feature: performing feature enhancement based on the difference features between it and the semantic features of its historical adjacent frames to generate enhanced features; selectively modulating the enhanced features using the spatial semantic weights corresponding to the frame semantic features to increase the contribution of static regions to state updates and suppress the interference of dynamic regions on state updates, obtaining a first updated state; fusing the output features corresponding to the first updated state with the frame semantic features to generate corresponding temporal features; and calculating the pose information of the target object based on the generated frame temporal features. This invention can achieve higher accuracy and stronger robustness in pose estimation in complex dynamic environments.

Description

Technical Field

[0001] This invention belongs to the field of robot autonomous localization and navigation, and more specifically, relates to a pose estimation method, device, and medium based on semantic visual odometry. Background Technology

[0002] Visual odometry is a key component of visual simultaneous localization and mapping (SMR) systems. It primarily estimates the camera's trajectory in three-dimensional space by analyzing a series of consecutive images. This technology has been widely applied in mobile robotics, autonomous navigation for drones, augmented reality, and virtual reality. Traditional visual odometry methods typically use RGB images as the main input, extracting image information through handcrafted features or deep learning networks to estimate pose. However, RGB images are dependent on ambient lighting, and their pixels are easily affected by changes in illumination, shadows, and low light levels, leading to unstable feature extraction and reduced accuracy and robustness of pose estimation. This problem is particularly pronounced in complex scenes with strong light variations or sparse textures.

[0003] With the development of deep learning technology, semantic information has been introduced into visual perception tasks. Semantic images, through category-level annotation of scenes, possess stronger structural expressive power and illumination robustness, providing more stable feature representations. Therefore, pose estimation based on semantic information helps improve the stability of visual odometry in complex environments. Simultaneously, end-to-end deep learning methods can directly learn the mapping relationship from input to pose, reducing reliance on manual design. Therefore, combining semantic information with an end-to-end framework to construct a visual odometry method using semantic images as input has significant research and application value.

[0004] Existing semantic visual odometry systems struggle to achieve high-precision and robust camera pose estimation in complex dynamic scenes. The main reasons for this can be summarized in three aspects. First, the utilization of semantic information is limited, failing to fully leverage its advantages: Existing methods typically use semantic information as auxiliary information, fusing it with RGB images for dynamic object removal or feature selection, rather than directly as core input in the pose estimation process. This approach fails to fully utilize the advantages of semantic information in structural representation stability and resistance to illumination interference, resulting in limited overall performance improvement in complex environments. Second, there is a lack of effective modeling of temporal information, leading to insufficient adaptability to dynamic scenes: While semantic images can provide high-level semantic structures, existing methods mostly process frame-by-frame or local information, lacking the ability to model cross-frame temporal correlations. In scenes with dynamic objects or rapid movement, semantic inconsistencies or matching errors are easily generated, affecting the continuity and stability of pose estimation. Third, some deep learning-based semantic visual odometry methods rely on complex network structures or multi-module fusion strategies, resulting in high computational overhead and making it difficult to achieve efficient operation on resource-constrained platforms (such as drones or embedded devices), thus limiting their practical application scope. Summary of the Invention

[0005] The main objective of this invention is to provide a pose estimation method, device, and medium based on semantic visual odometry to overcome the shortcomings of the prior art.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] The first aspect of this invention provides a pose estimation method based on semantic visual odometry, comprising: preprocessing an original image sequence of a target object to generate a semantic image sequence; inputting the semantic image sequence into a lightweight feature extraction network to extract continuous frame semantic features that can describe spatial structure information; wherein the lightweight feature extraction network includes multiple cascaded sub-networks, each sub-network including several layers of gated convolutional neural networks and a Mamba temporal modeling network connected in sequence; for each frame semantic feature: performing feature enhancement based on the difference features between it and the semantic features of historical adjacent frames to generate enhanced features; selectively modulating the enhanced features using the spatial semantic weights corresponding to the frame semantic features to increase the contribution of static regions to state updates and suppress the interference of dynamic regions on state updates, to obtain a first updated state; fusing the output features corresponding to the first updated state with the frame semantic features to generate corresponding temporal features; and calculating the pose information of the target object based on the generated frame temporal features.

[0008] Preferably, the processing steps in the gated convolutional neural network include: extracting features from the received first input features to obtain first extracted features; using the gate weights corresponding to the first input features to perform gated modulation on the first extracted features to generate first modulated features; and fusing the first modulated features and the first input features through channel mapping and residual connections to obtain first output features.

[0009] Preferably, the processing steps in the Mamba temporal modeling network include: converting the received second input features into a sequence form to obtain second sequence features; for each feature in the second sequence features: performing linear projection on the feature, selectively modulating the feature obtained by linear projection to obtain a second update state; reconstructing the output features corresponding to each second update state into spatial features in a temporal sequence; and connecting the second input features and the spatial features to obtain a second output feature.

[0010] Preferably, the second update state is:

[0011] ; in, , These are the second update states corresponding to the t-th and t-1-th features in the second sequence, respectively. For the t-th feature in the second sequence features, For the input of the relevant first parameter function, For the input of the relevant second parameter function, The features are obtained by linear projection.

[0012] Preferably, the calculation process of the sub-network is as follows:

[0013] ; in, , These are the outputs of the (k+1)th and kth subnetworks in a cascaded network, respectively. K is the total number of subnetworks. This represents the processing procedure in the gated convolutional neural network. This represents the processing procedure in the Mamba time-series modeling network.

[0014] Preferably, the enhanced feature is:

[0015] ;

[0016] in, For the semantic features of frame t The corresponding enhancement features, for The flattened sequence form, This is the differential adjustment coefficient. for semantic features of frame t-1 The sequence form after flattening the difference features between them.

[0017] Preferably, the first update state is:

[0018] ;

[0019] in, , Semantic features of frame t Semantic features of frame t-1 The corresponding first update status, for The corresponding enhancement features, for Features after linear mapping for The corresponding spatial semantic weight flattened sequence form, This represents element-wise multiplication. For the input of the relevant first parameter function, The second parameter function is the input.

[0020] Preferably, the pose information of the target object is calculated based on the generated temporal features of each frame, specifically including: fusing and optimizing the temporal features of the current frame with the temporal features of historical frames to generate corresponding fused and optimized features; inputting the fused and optimized features into a pose regression network to output the relative poses between adjacent frames; accumulating the relative poses to calculate the complete motion trajectory of the target object, wherein the pose information includes the complete motion trajectory.

[0021] A second aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the attitude estimation method based on semantic visual odometry as described above.

[0022] A third aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the attitude estimation method based on semantic visual odometry as described above.

[0023] Compared with existing technologies, the advantages of this invention are as follows: This invention provides a pose estimation method, device, and medium based on semantic visual odometry. By constructing a visual odometry framework with semantic images as the core input, and introducing a semantically guided selective state update mechanism and a cross-frame differential enhancement strategy during temporal modeling, it effectively overcomes the problems of traditional methods such as strong dependence on RGB information, susceptibility to changes in illumination, and unstable modeling in dynamic scenes. Specifically, semantic information provides a more stable structural expression, ensuring consistency of features under complex illumination conditions; a lightweight CNN-Mamba backbone network is used to guarantee the real-time performance of the algorithm; the cross-frame differential mechanism explicitly characterizes scene motion changes, improving the model's sensitivity to pose changes; simultaneously, by introducing semantic weight modulation into the state space model, it achieves enhanced modeling of static regions and effective suppression of dynamic interference, thereby improving the effectiveness of temporal feature modeling causally; based on the above technical solutions, this invention can achieve higher accuracy and stronger robustness of pose estimation in complex dynamic environments while maintaining computational efficiency, and has good practical application value. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a pose estimation method based on semantic visual odometry provided in an embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating the processing procedure of a sub-network provided in an embodiment of the present invention.

[0027] Figure 3 A flowchart illustrating the processing procedure of a time-series modeling network provided in an embodiment of the present invention.

[0028] Figure 4 A block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0029] In view of the shortcomings of the prior art, the inventors of this invention, through long-term research and extensive practice, have proposed the technical solution of this invention. The following will further explain and illustrate this technical solution, its implementation process, and its principles.

[0030] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0031] Furthermore, in the description of this invention, it should be understood that the terms "upper," "lower," "inner," "outer," "horizontal," "vertical," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0032] In the description of this specification, the references to terms such as "an embodiment," "a particular embodiment," or "the embodiment" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0033] Figure 1 A flowchart illustrating the pose estimation method based on semantic visual odometry provided in this embodiment of the invention. (See also...) Figure 1 , combined Figures 2-3 The pose estimation method based on semantic visual odometry provided in the embodiments of the present invention will be described in detail. The method includes operation S100-operation S400.

[0034] Operation S100 preprocesses the original image sequence of the target object to generate a semantic image sequence.

[0035] Operation S200 inputs the semantic image sequence into the lightweight feature extraction network to extract continuous frame semantic features that can describe spatial structure information; wherein, the lightweight feature extraction network includes multiple cascaded sub-networks, and the sub-networks include several layers of gated convolutional neural networks and Mamba temporal modeling networks connected in sequence.

[0036] Operation S300: For each frame semantic feature, enhance its features based on the difference between its semantic features and those of historical adjacent frames to generate enhanced features; selectively modulate the enhanced features using the spatial semantic weights corresponding to the semantic features of the frame to increase the contribution of static regions to state updates and suppress the interference of dynamic regions on state updates, thereby obtaining the first updated state; fuse the output features corresponding to the first updated state with the semantic features of the frame to generate the corresponding temporal features.

[0037] Operate S400 to calculate the target's attitude information based on the generated temporal features of each frame.

[0038] The pose estimation method based on semantic visual odometry provided in this invention overcomes the problems of traditional methods, such as strong dependence on RGB information, susceptibility to changes in illumination, and unstable modeling in dynamic scenes, by constructing a visual odometry framework with semantic images as the core input and introducing a semantically guided selective state update mechanism and a cross-frame differential enhancement strategy during temporal modeling. The target objects in this method include, but are not limited to, robots and handheld devices.

[0039] Preferably, operation S100 specifically includes: acquiring a continuous temporal sequence of original images of the target object, performing scale normalization and data standardization on the original image sequence, and then inputting it into a semantic segmentation model, which generates a corresponding semantic image sequence.

[0040] In a preferred embodiment, the processing steps in the gated convolutional neural network include: extracting features from the received first input features to obtain first extracted features; using the gate weights corresponding to the first input features to perform gated modulation on the first extracted features to generate first modulated features; and fusing the first modulated features and the first input features through channel mapping and residual connections to obtain first output features.

[0041] In a preferred embodiment, the processing steps in the Mamba temporal modeling network include: converting the received second input features into a sequence form to obtain second sequence features; for each feature in the second sequence features: performing linear projection on the feature, selectively modulating the feature obtained by linear projection to obtain a second update state; reconstructing the output features corresponding to each second update state into spatial features in a temporal sequence; and connecting the second input features and the spatial features to obtain a second output feature.

[0042] In a preferred embodiment, the second update state is:

[0043] ;

[0044] in, , These represent the second update states corresponding to the t-th and t-1-th features in the second sequence, respectively. For the t-th feature in the second sequence features, For the input of the relevant first parameter function, For the input of the relevant second parameter function, The features are obtained by linear projection.

[0045] In a preferred embodiment, the calculation process of the sub-network is as follows:

[0046] ;

[0047] in, , These are the outputs of the (k+1)th and kth subnetworks in a cascaded network, respectively. K is the total number of subnetworks. This represents the processing steps in a gated convolutional neural network. This represents the processing steps in the Mamba time-series modeling network.

[0048] See Figure 2 This invention specifically describes the working process of the lightweight feature extraction network in the S200. It proposes a lightweight feature extraction network combining CNN and Mamba. The lightweight feature extraction network consists of multiple stages, each stage being a sub-network. Each stage contains several layers of gated convolutional neural networks (Gated CNN) and one layer of Mamba temporal modeling network, designed to improve feature representation capabilities while ensuring computational efficiency.

[0049] Gated convolutional neural networks consist of a feature extraction branch and a gating weight branch. The input features are defined. First, regarding the input features... Perform the following processing:

[0050] ;

[0051] Secondly, gating weights are generated through gating weight branches. (Spatial semantic weights):

[0052] ;

[0053] Then, for features Perform gated modulation:

[0054] ;

[0055] Finally, the output features are obtained through channel mapping and residual connections. :

[0056] ;

[0057] in, This represents a 3×3 convolution kernel. Represents a non-linear activation function. The range of values ​​is , This represents a 1×1 convolution kernel.

[0058] The shallow Mamba temporal modeling network is represented as follows. The output features of the gated convolutional neural network are converted into sequence form, represented as:

[0059] ;

[0060] in, For the t-th element in the sequence The position, its input is The computation process of the Mamba time series modeling network is as follows.

[0061] First, perform linear projection:

[0062] ;

[0063] Then, the status is updated:

[0064] ;

[0065] Next, calculate the output. :

[0066] ;

[0067] Then, all time-series outputs are reconstructed into spatial features. :

[0068] ;

[0069] Finally, the output of the Mamba temporal modeling network is obtained through residual connections. :

[0070] ;

[0071] in, It is in a hidden state. This is the output mapping function.

[0072] For the Each stage, and its overall calculation process, is represented as follows:

[0073] ;

[0074] Multiple cascaded stages form a complete backbone network, enabling feature representation and modeling from shallow to deep. Specifically, the gated convolutional neural network extracts local spatial structure features through 3×3 convolutions and generates gate weights using 1×1 convolutions, selectively modulating feature responses at different locations and channels to highlight stable semantic regions related to pose estimation and reduce the impact of redundant or unstable information. The Mamba temporal modeling network models the serialized features through an input-related state update mechanism, capturing spatial locations and dependencies between continuous features with low computational overhead. Since the gated convolutions mainly use 1×1 and 3×3 small convolution kernels, the number of parameters and computational cost are low; the computational complexity of the Mamba structure increases approximately linearly with the sequence length, making it more suitable for lightweight deployment compared to self-attention mechanisms. Through the above lightweight feature extraction network structure design, this invention can effectively model spatial features and long-range dependencies while maintaining low computational complexity, thereby improving the accuracy and robustness of pose estimation in visual odometry tasks.

[0075] In a preferred embodiment, the enhanced feature generated in operation S300 is:

[0076] ;

[0077] in, For the semantic features of frame t The corresponding enhancement features, for The flattened sequence form; This is the differential adjustment coefficient, used to control the intensity of motion information introduction; for semantic features of frame t-1 The sequence form after flattening the difference features between them.

[0078] In a preferred embodiment, the first update state obtained in operation S300 is:

[0079] ;

[0080] in, , Semantic features of frame t Semantic features of frame t-1 The corresponding first update status, for The corresponding enhancement features, for Features after linear mapping for The corresponding spatial semantic weight flattened sequence form, This represents element-wise multiplication. For the input of the relevant first parameter function, The second parameter function is the input.

[0081] The following combination Figure 3 The processing steps in operation S300 are described in detail. Operation S300 can be implemented using a temporal modeling network. To address the problems of existing methods' inability to distinguish between dynamic and static regions and the instability of temporal modeling, this invention proposes a semantically guided selective state-space temporal modeling network (implementation process as follows). Figure 3 As shown in the figure, and combined with the cross-frame differential enhancement mechanism, the specific implementation process is as follows.

[0082] (1) Semantic feature sequence construction.

[0083] Let the semantic features corresponding to the input image in frame t be... Where C represents the number of channels, and H and W represent [other parameters]. To facilitate time series modeling, the spatial features are flattened into a sequence form:

[0084] ;

[0085] in, This sequence represents the feature distribution of the current frame in the spatial dimension and serves as the basic input for subsequent temporal modeling.

[0086] (2) Construction of cross-frame differential features.

[0087] To enhance the model's ability to perceive motion information, differential features between adjacent frames are introduced. Specifically, the feature differences between the current frame and the previous frame are calculated. :

[0088] ;

[0089] The difference features are also flattened into sequence form:

[0090] ;

[0091] Constructing enhanced input features (enhanced features) ):

[0092] ;

[0093] This step allows for explicit modeling of changing regions in the scene, enabling the model to focus more on motion-related information and thus improving the accuracy of pose estimation.

[0094] (3) Semantic guidance weight (spatial semantic weight) generation mechanism.

[0095] To distinguish between dynamic and static structural regions, this invention utilizes semantic features to generate a spatial semantic weight map. Specifically, the semantic features are subjected to convolutional transformation and normalized using the Sigmoid function:

[0096] ;

[0097] The kernel size k can be set according to requirements. The weight map... Expand into sequence form:

[0098] ;

[0099] Weighted graphs are used to represent the importance of different spatial locations, where stable static regions typically have higher weights, while dynamic regions have lower weights.

[0100] (4) Temporal modeling based on selective state-space model.

[0101] Enhanced feature sequences Input time-series modeling network. This invention uses an improved state-space model (Mamba structure) for modeling and introduces a semantically guided modulation mechanism during state update.

[0102] First, perform a linear mapping on the input features:

[0103] ;

[0104] Then the status is updated:

[0105] ;

[0106] By introducing spatial semantic weights This allows for selective modulation of the input information, enabling static regions to contribute more to state updates while effectively suppressing interference from dynamic regions. Next, the output features are calculated. :

[0107]

[0108] (5) Temporal feature reconstruction and fusion.

[0109] Output the sequence Restored to spatial structure :

[0110] ;

[0111] Then, with the original semantic features To merge:

[0112]

[0113] The embodiments of the present invention achieve effective fusion of spatial features and temporal features through residual connection.

[0114] The temporal modeling network of this invention does not directly use a conventional state-space model to uniformly update the input sequence. Instead, it introduces a cross-frame differential enhancement mechanism before state updates and a semantically guided spatial weight modulation mechanism during the state update process. Specifically, cross-frame differential features are used to explicitly represent the semantic structural differences between the current frame and historical adjacent frames, solving the problems of insufficient modeling of inter-frame motion changes and weak perception of pose changes in existing methods. This enables the temporal modeling network to more sensitively capture scene changes caused by camera motion. Spatial semantic weights are used to perform position-level modulation on the enhanced features, making stable static semantic regions contribute more to state updates, while the influence of dynamic targets, occluded regions, or semantically unstable regions is relatively weakened. This solves the problems of existing methods in distinguishing stable structures from dynamic interference in dynamic scenes and the susceptibility of temporal feature updates to moving objects. Through the above design, this invention can obtain more stable temporal feature representations and improve the accuracy and robustness of relative pose estimation between adjacent frames.

[0115] In a preferred embodiment, operation S400 specifically includes the following sub-operations S410-S430.

[0116] In sub-operation S410, the temporal features of the current frame and the temporal features of historical frames are fused and optimized to generate corresponding fused and optimized features.

[0117] Specifically, the temporal features of the current frame are fused with the temporal features of historical frames, and key region features are enhanced through attention mechanisms or weighting strategies to suppress the influence of dynamic interference regions.

[0118] In sub-operation S420, the fused and optimized features are input into the pose regression network, which outputs the relative pose between adjacent frames. The relative pose includes translation vectors and rotation parameters.

[0119] In sub-operation S430, the relative pose is accumulated, and the complete motion trajectory of the target object (e.g., 6-DOF motion parameters) is calculated. The attitude information includes the complete motion trajectory.

[0120] The pose estimation method based on semantic visual odometry provided in this invention has several advantages. First, it employs end-to-end semantic odometry, using semantic images instead of traditional RGB images as visual odometry input, which improves feature stability and resistance to illumination interference, forming the basis for achieving highly robust pose estimation in complex scenes. Second, it designs a lightweight CNN-Mamba backbone structure, combining gated convolutional networks with shallow Mamba modules to achieve efficient modeling of spatial features and long-range temporal dependencies while ensuring computational efficiency, making it suitable for real-time or embedded visual odometry applications. Third, it designs a semantically guided selective state update mechanism, introducing spatial weight modulation based on semantic features into the temporal modeling network to achieve enhanced modeling of static structural regions and suppression of interference in dynamic regions, thereby improving the effectiveness of temporal features and pose continuity.

[0121] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the attitude estimation method based on semantic visual odometry as described in any of the above embodiments.

[0122] Figure 4 This illustration shows a more specific hardware structure diagram of an electronic device provided in this embodiment. The device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, memory 420, input / output interface 430, and communication interface 440 are interconnected internally via the bus 450.

[0123] The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0124] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0125] Input / output interface 430 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0126] The communication interface 440 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0127] Bus 450 includes a pathway for transmitting information between various components of the device, such as processor 410, memory 420, input / output interface 430, and communication interface 440.

[0128] It should be noted that although the above-described device only shows the processor 410, memory 420, input / output interface 430, communication interface 440, and bus 450, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0129] The electronic devices described in the above embodiments are used to implement the attitude estimation method based on semantic visual odometry as described in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0130] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the attitude estimation method based on semantic visual odometry as described in any of the above embodiments.

[0131] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0132] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the attitude estimation method based on semantic visual odometry as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0133] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of the invention as described above, which are not provided in the details for the sake of brevity.

[0134] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of the invention, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of the invention, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of the invention will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that the embodiments of the invention may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0135] Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0136] It should be understood that the above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A pose estimation method based on semantic visual odometry, characterized in that, include: The original image sequence of the target object is preprocessed to generate a semantic image sequence; The semantic image sequence is input into a lightweight feature extraction network to extract continuous frame semantic features that can describe spatial structure information; wherein, the lightweight feature extraction network includes multiple cascaded sub-networks, and the sub-networks include several layers of gated convolutional neural networks and Mamba temporal modeling networks connected in sequence; For each frame semantic feature: enhance the feature based on the difference between the semantic features of the frame and the semantic features of the historical adjacent frames to generate enhanced features; selectively modulate the enhanced features using the spatial semantic weights corresponding to the semantic features of the frame to increase the contribution of static regions to state updates and suppress the interference of dynamic regions to state updates, to obtain the first update state; fuse the output features corresponding to the first update state with the semantic features of the frame to generate the corresponding temporal features. The pose information of the target object is calculated based on the generated temporal features of each frame.

2. The pose estimation method based on semantic visual odometry according to claim 1, characterized in that, The processing steps in the gated convolutional neural network include: The first input features received are used to extract features, resulting in the first extracted features; The first extracted feature is gated and modulated using the gate weights corresponding to the first input feature to generate the first modulated feature. The first modulation feature and the first input feature are fused by channel mapping and residual connection to obtain the first output feature.

3. The pose estimation method based on semantic visual odometry according to claim 1, characterized in that, The processing steps in the Mamba time-series modeling network include: The received second input features are converted into sequence form to obtain the second sequence features; For each feature in the second sequence features: perform linear projection on the feature, and use the feature obtained by linear projection to selectively modulate the feature to obtain the second update state; The output features corresponding to each of the second update states are reconstructed into spatial features in a temporal sequence. The second input feature and the spatial feature are connected to obtain the second output feature.

4. The pose estimation method based on semantic visual odometry according to claim 3, characterized in that, The second update status is: ； in, , These are the second update states corresponding to the t-th and t-1-th features in the second sequence, respectively. For the t-th feature in the second sequence features, For the input of the relevant first parameter function, For the input of the relevant second parameter function, The features are obtained by linear projection.

5. The pose estimation method based on semantic visual odometry according to any one of claims 1-4, characterized in that, The calculation process of the sub-network is as follows: ； in, , These are the outputs of the (k+1)th and kth subnetworks in a cascaded network, respectively. K is the total number of subnetworks. This represents the processing procedure in the gated convolutional neural network. This represents the processing procedure in the Mamba time-series modeling network.

6. The pose estimation method based on semantic visual odometry according to claim 1, characterized in that, The enhanced feature is: ； in, For the semantic features of frame t The corresponding enhancement features, for The flattened sequence form, This is the differential adjustment coefficient. for semantic features of frame t-1 The sequence form after flattening the difference features between them.

7. The pose estimation method based on semantic visual odometry according to claim 1, characterized in that, The first update status is: ； in, , Semantic features of frame t Semantic features of frame t-1 The corresponding first update status, for The corresponding enhancement features, for Features after linear mapping for The corresponding spatial semantic weight flattened sequence form, This represents element-wise multiplication. For the input of the relevant first parameter function, The second parameter function is the input.

8. The pose estimation method based on semantic visual odometry according to claim 1, characterized in that, Based on the generated temporal features of each frame, the pose information of the target object is calculated, specifically including: The temporal features of the current frame are fused and optimized with the temporal features of historical frames to generate corresponding fused and optimized features; The fused and optimized features are input into the pose regression network, which outputs the relative poses between adjacent frames. By accumulating the relative pose, the complete motion trajectory of the target object is calculated, and the pose information includes the complete motion trajectory.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the attitude estimation method based on semantic visual odometry as described in any one of claims 1-8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the attitude estimation method based on semantic visual odometry as described in any one of claims 1-8.

Images