Pose estimation method and device based on visual odometer, equipment and medium
Through a deep learning method driven by multi-head attention and self-attention mechanisms, the problems of poor generalization and motion consistency feature extraction in visual odometry are solved, and accurate estimation of camera pose is achieved.
Patent Information
- Application Number
- CN202510872139.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-26
- Publication Date
- 2025-10-10
AI Technical Summary
At present, the visual odometry calculation method based on deep learning has problems such as poor generalization and difficulty in extracting motion consistency features, which affects the positioning effect of the visual SLAM system.
A deep learning method based on multi-head attention mechanism and self-attention mechanism is adopted to extract image features through optical flow data, and feature optimization is performed in combination with motion consistency constraints, and finally pose fitting is performed.
It improves the pose estimation accuracy and generalization ability of the visual odometry, solves the problem of difficulty in extracting motion consistency features, and realizes accurate estimation of the camera pose.
Smart Images
Figure CN120765740A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of pose estimation, and in particular to a pose estimation method, device, equipment and medium based on visual odometry. Background Art
[0002] In visual SLAM systems, the visual odometry (VoM) is the most important module for predicting camera motion and pose changes. It can predict the motion of unmanned control systems simply by analyzing consecutive image frames captured by the camera, making it a key module in visual SLAM systems. The VoM primarily calculates the camera displacement between adjacent image frames, so it can only optimize pose predictions locally. However, current deep learning-based VoM calculation methods suffer from poor generalization and difficulty extracting motion consistency features. Summary of the Invention
[0003] The present invention provides a method, apparatus, device and medium for pose estimation based on visual odometry to achieve accurate estimation of camera pose.
[0004] According to one aspect of the present invention, a method for pose estimation based on visual odometry is provided, comprising:
[0005] Obtain the image to be processed captured by the camera, and extract the optical flow data of the image to be processed based on the optical flow method;
[0006] Based on the optical flow data, the multi-head attention mechanism and feature extraction module are used to extract features of the image to be processed, and the motion features to be optimized corresponding to the image to be processed are obtained;
[0007] According to the motion consistency constraint, the optimized features are optimized to obtain the target motion features corresponding to the image to be processed;
[0008] The pose fitting is performed based on the target running characteristics to obtain the corresponding pose estimation of the camera.
[0009] According to another aspect of the present invention, a posture estimation device based on visual odometry is provided, comprising:
[0010] The image processing module is used to obtain the image to be processed captured by the camera and extract the optical flow data of the image to be processed based on the optical flow method;
[0011] The feature extraction module is used to extract features of the image to be processed based on the optical flow data through the multi-head attention mechanism and the feature extraction module to obtain the motion features to be optimized corresponding to the image to be processed;
[0012] A feature optimization module is used to optimize the features to be optimized according to the motion consistency constraint to obtain the target motion features corresponding to the image to be processed;
[0013] The pose estimation module is used to perform pose fitting based on the target's operating characteristics to obtain the corresponding pose estimation of the camera.
[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0015] at least one processor;
[0016] and a memory communicatively connected to at least one processor; wherein,
[0017] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the pose estimation method based on visual odometry according to any embodiment of the present invention.
[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the pose estimation method based on visual odometry according to any embodiment of the present invention when executed.
[0019] The technical solution of the embodiment of the present invention fully utilizes the optical flow information of pixels in the input image sequence, extracts motion features on this basis, optimizes and filters the features through the motion consistency module, and finally completes the relative pose estimation task in a local range. It solves the problems of poor generalization and difficulty in extracting motion consistency features in the current deep learning-based visual odometer calculation method. Compared with the image sequence processing methods of other deep visual odometers, the present invention fully considers the geometric constraint relationship of the camera motion between the input image sequences, and preferentially extracts the optical flow data of the input image sequence before performing the next step of pose estimation. Through the analysis of geometric motion data, the present invention further filters the high-dimensional motion features of the camera through the motion consistency constraint module, and then fits the relative pose estimate of the camera.
[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1A flowchart of a method for posture estimation based on visual odometry provided in an embodiment of the present invention;
[0023] Figure 2 A schematic diagram of a PWC-Net model provided in an embodiment of the present invention;
[0024] Figure 3 Schematic diagram of the multi-head attention mechanism provided by an embodiment of the present invention;
[0025] Figure 4 LSTM-based RNN network architecture diagram provided by an embodiment of the present invention;
[0026] Figure 5 A diagram showing the stacked LSTM and its time-varying expansion and internal LSTM unit structure provided in an embodiment of the present invention;
[0027] Figure 6 A framework diagram of a motion consistency constraint module driven by a self-attention mechanism according to an embodiment of the present invention;
[0028] Figure 7 An algorithm diagram of the self-attention mechanism provided by an embodiment of the present invention;
[0029] Figure 8 The RCNN network model diagram provided by the embodiment of the present invention;
[0030] Figure 9 A schematic structural diagram of a posture estimation device based on visual odometry provided by an embodiment of the present invention;
[0031] Figure 10 A schematic diagram of the structure of an electronic device for implementing a pose estimation method based on visual odometry according to an embodiment of the present invention. DETAILED DESCRIPTION
[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.
[0033] It is to be understood that the terminology "first", "second" and the like used in the specification and the claims of the application as well as the foregoing drawings is merely intended to distinguish between similar objects and not necessarily for describing a special sequential order. It is to be understood that the data so used in the specification and the claims of the application are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of use in other sequences other than the ones explicitly described or otherwise shown as the primary flow. Furthermore, the term "comprising" and "including" as well as their grammatical variations, used throughout the specification and in the claims of the application, are each intended to signify an open group such that also embodiments including, for example, only those further steps or units are considered to be within the scope of the application.
[0034] At present, with the deepening of the intersection and integration of the traditional automobile industry field and the computer science field, how to realize the rapid, safe and reliable automatic driving of the automobile has become a hot topic in the field of artificial intelligence in the world today. The broad market and precise technical requirements have greatly attracted the interest of scientists and engineers in the field. The key to automatic driving technology lies in how the vehicle autonomously constructs a model of the surrounding environment and accurately obtains its own position information in the constructed model, so simultaneous localization and mapping (SLAM) naturally attracts widespread attention. Today, SLAM positioning technology has developed in the direction of various sensors, among which the visual SLAM algorithm uses various types of cameras as the main sensor, which can still guarantee relatively stable positioning effect in areas where satellite signals are relatively weak or the environment is relatively harsh. In addition, cameras have the advantages of relatively low cost, rich surrounding environment information acquisition, and convenient installation, so the visual SLAM technology has become an important hot spot in the field of SLAM. A key problem of traditional visual SLAM technology is the need to rely on the effect of camera calibration, which is quite complex and tedious. The effect of feature point tracking and detection in the front-end visual odometry (VO) directly affects the final positioning effect of the visual SLAM system. In recent years, the visual odometry based on deep learning algorithm provides a more accurate implementation scheme. However, the current visual odometry algorithm based on deep learning has the problems of poor generalization, difficulty in extracting motion consistency features, etc. Therefore, the present application researches a deep learning visual odometry algorithm based on motion consistency self-attention mechanism.
[0035] Figure 1A flowchart of a pose estimation method based on a visual odometer is provided for an embodiment of the present application. The embodiment can be applicable to the estimation of camera pose. The method can be performed by a pose estimation device based on a visual odometer. The device can be implemented in the form of hardware and / or software. The device can be configured in a vehicle. As shown in Figure 1 , the method specifically includes the following steps:
[0036] S110, obtaining an image to be processed collected by a camera, and extracting optical flow data of the image to be processed based on an optical flow method.
[0037] The optical flow data is information describing the motion trajectory and speed of a pixel point over time in consecutive image frames.
[0038] It should be noted that the optical flow method can analyze the motion process of a pixel to obtain the motion of the camera, which to some extent expresses the geometric features between image sequences captured by the camera.
[0039] Although FlowNet2.0 uses a neural network to improve the operation efficiency, it is difficult to meet the real-time calculation due to the relatively low sampling frequency of the camera. After screening and considering several optical flow extraction networks, the PWC-Net model is finally selected as the optical flow extraction module, as shown in Figure 2 .
[0040] In some embodiments, the optical flow data of the image to be processed is extracted based on the optical flow method, including processing the image to be processed based on an optical flow data extraction module to obtain the optical flow data.
[0041] Specifically, the PWC-Net network adopts a pyramid image processing structure, and its calculation process includes feature extraction, cost volume construction, optical flow data and context network, etc. The parameter amount of PWC-Net is reduced by about 17 times compared with FlowNet2.0. The formula for extracting the optical flow data is:
[0042]
[0043] wherein I t-1 and I t represent consecutive images collected by the camera at adjacent time t-1 and t, is the optical flow data calculated by the function ; C, H and W represent the channel number, length and width of the optical flow data, and the channel number C=2 represents the motion conditions of the pixel in the horizontal and vertical coordinate directions.
[0044] S120, extracting features of the image to be processed based on a multi-head attention mechanism and a feature extraction module according to the optical flow data, to obtain the motion features to be optimized corresponding to the image to be processed.
[0045] Among them, the feature extraction module is used to extract the high-dimensional motion features of the image, and the motion features to be optimized are the features extracted by this module.
[0046] Specifically, the optical flow data can be combined to perform feature extraction on the image to be processed through a multi-head attention mechanism and a feature extraction module to obtain the motion features to be optimized of the image to be processed.
[0047] In some embodiments, the feature extraction module includes a spatial feature extraction submodule and a temporal feature extraction submodule, and performs feature extraction on the image to be processed through a multi-head attention mechanism and the feature extraction module to obtain motion features to be optimized corresponding to the image to be processed, including: inputting optical flow data and the image to be processed into the spatial feature extraction submodule to obtain the spatial features of the image to be processed; inputting the spatial features into the temporal feature extraction submodule to obtain the temporal features of the image to be processed; processing the image to be processed through the multi-head attention mechanism to obtain the multi-head attention mechanism features corresponding to the image to be processed; fusing the spatial features, temporal features and multi-head attention mechanism features to obtain the motion features to be optimized.
[0048] Among them, the spatial feature extraction submodule can be a convolutional neural network (CNN), and the temporal feature extraction submodule can be a recurrent neural network (RNN).
[0049] In this embodiment, a cascaded model of two neural networks is used to extract high-dimensional motion features from images. First, a convolutional neural network is used to extract features from the captured image, fully extracting information such as the spatial correlation reflected in the optical flow data. Then, a recurrent neural network is used to extract temporal correlation information from the image sequence. The implementation of this embodiment is described in detail below.
[0050] First, directly use the cropped monocular RGB image as the sequence input. For a given image Its spatial resolution is H×W, the number of channels is C, and the plane of the input image x is cut into several image blocks with a size of P×P. N represents the number of image blocks (i.e., the length of the input sequence), which is specifically expressed as follows:
[0051]
[0052] To train the model, the model linearly projects the image patch vector x p Mapped in the D-dimensional hidden state space. For the spatial information and position information of the image, the model uses a specific encoding embedding, which can retain the position information while learning the spatial information. represents the image block embedding vector, Indicates the image block position embedding, which is specifically expressed as follows:
[0053]
[0054] In order for the entire network to extract richer feature information, a multi-head self-attention mechanism has been gradually developed based on the self-attention mechanism and is widely used in the field of machine vision. The three variables Key, Value, and Query are first linearly transformed and then used as input values to perform h operations in the scaled dot product Attention. Each time the Key, Value, and Query are linearly transformed, the parameter matrix of the linear transformation will also be transformed. Then, the results of the scaled dot product Attention after h operations are linked and finally linearly transformed again to obtain the results of the multi-head self-attention mechanism. The multi-head attention mechanism is as follows: Figure 3 As shown:
[0055]
[0056] Among them, Q represents Query, i.e., the query vector sequence; K represents Key, i.e., the key vector sequence; V represents Value, i.e., the value vector sequence; d k Represents the dimension of the vector.
[0057] The present invention integrates a CNN network model into the feature extraction module. CNN is first used to extract features from the original image, generating a feature map as input. The extracted latent vector is then embedded into the multi-head self-attention module, and the resulting image feature information is used as the module's output.
[0058] S t =f(S t-1 ,Y t-1 ,C t )...................................(6)
[0059]
[0060] e tj =g(S t-1 ,h j ).......................................(9)
[0061] Where h represents the input latent vector, f represents the activation function; Ct represents the attention distribution. Assuming that a relevant query q is given, it represents the degree of attention paid to the t-th information; α tjThe attention represents the attention of each input acquisition. After introducing the self-attention mechanism, the model can better observe the global implementation. The scoring function in the model is generally composed of the following models: dot product model, additive model, scaled dot product or bilinear model, as shown below:
[0062]
[0063] In the above formula, W, U, V T represent the learnable network parameters, and d is the dimension of the input information.
[0064] Since the CNN network has obvious defects in learning image information at different times, in order to solve this problem, we serially stacked the RNN recurrent network in the model, so that the network can learn the feature information between image frames at different times, and through the self-attention mechanism, the image information at adjacent times is linked together, effectively avoiding the problem of gradient vanishing. In order to make the system can also effectively analyze the correlation between image frames on long track, the model uses the long short-term memory network (LSTM) in the RNN network. The RNN network used in the invention is stacked with three LSTM layers, and the input of one LSTM is used as the hidden state of another LSTM. The RNN network architecture based on LSTM is as shown in Figure 4 Each LSTM layer contains 800 hidden states.
[0065] The stacked LSTM and its time-varying expansion form and internal LSTM unit structure diagram are as shown in Figure 5 From the expansion form of the LSTM, it can be seen that each LSTM unit is associated with a time step. Given the input x t , the hidden state h t-1 and the memory unit m t-1 of the previous LSTM unit at time t, the update rule of the LSTM at time step t is as follows:
[0066] i t =σ(W xi x t +W hi h t-1 +b i )............................(14)
[0067] f t =σ(W xf x t +W hf h t-1 +b f )...........................(15)
[0068] g t =tanh(W xg x t +W hg h t-1 +b g ).........................(16)
[0069] m t =f t ⊙m t-1 +i t ⊙g t ............................(17)
[0070] o t =σ(W xo x t +W ho h t-1 +b o )...........................(18)
[0071] h t =o t ⊙tanh(m t )................................(19)
[0072] Where ⊙ represents the element-wise product of two vectors; σ represents the sigmoid nonlinear function; tanh is the hyperbolic tangent nonlinear function; W represents the corresponding weight matrix; b represents the bias vector; i t 、f t 、g t 、m t and o t They correspond to the input gate, forget gate, input modulation gate, memory unit and output gate at time t respectively.
[0073] The LSTM unit processes feature information at the same time as the image sequence input to the visual odometry is acquired. In this visual odometry framework, this feature extraction module comprises two LSTM layers, each with 1024 dimensions, and both employ the nonlinear ReLU activation function. S130 optimizes the features to be optimized based on motion consistency constraints to obtain target motion features corresponding to the image to be processed.
[0074] In some embodiments, the optimization of the to-be-optimized feature according to the motion consistency constraint obtains a target motion feature corresponding to the to-be-processed image, and the optimization of the to-be-optimized feature according to the motion consistency constraint obtains a target motion feature corresponding to the to-be-processed image.
[0075] The target motion feature refers to a feature obtained by optimizing the to-be-optimized motion feature in a local range by the motion consistency constraint module.
[0076] In order to make the pose estimation of the visual odometry more accurate, the algorithm adds a motion consistency constraint module to optimize the motion feature in a local range. Since the camera is loaded on the vehicle for motion capture, the process has motion continuity, and the probability of discrete sudden motion is very small. Therefore, in a local range, the continuity information of the motion can be extracted, the consistency of the motion is modeled, and then the features of the camera constrained in the local range are screened to achieve the goal of optimizing the camera pose estimation.
[0077] In order to achieve this function, the motion feature screening algorithm driven by the self-attention mechanism is still used, which not only makes each time's motion retain its own independent feature, but also emphasizes the motion consistency information contained in these independent features. The self-attention mechanism has the characteristic of focusing on the data itself, so it can adaptively calculate the value of each parameter in the attention mask. The algorithm reduces the weight of outliers and noise to emphasize important motion features, so that the model can construct relatively important features to complete pose estimation. The feature optimization based on the self-attention mechanism used by the present application can be represented as:
[0078]
[0079] Wherein represents the relevant learning parameters in the motion consistency constraint module. Based on the guidance of the self-attention mechanism, the motion feature in the feature extraction module can be converted into the corresponding feature with motion consistency constraint The self-attention mechanism of the algorithm mainly reflects in the following two aspects, first, the feature at time t is constrained by the feature at the previous time t-1 ; in addition, the LSTM unit stores many hidden states, and their corresponding local motion context information also has a constraint effect on Under the joint guidance and supervision of these two information, the motion consistency constraint module can map the feature to a feature containing consistency information at the pixel level
[0080] Unlike the distillation method of the common attention network SENet model, the self-attention mechanism module proposed in the application is for extracting and modeling the consistency features of camera motion, and the extracted motion features are further screened and optimized in the time dimension. The framework of the motion consistency constraint module driven by the self-attention mechanism proposed in this paper is as shown in Figure 6 , wherein the self-attention mechanism algorithm in the core of the module is as shown in Figure 7 .
[0081] S140, pose fitting based on target running features to obtain the pose estimation corresponding to the camera.
[0082] In some embodiments, the pose fitting based on the target running features to obtain the pose estimation corresponding to the camera comprises: pose fitting based on a two-layer LSTM network and a two-layer fully connected network to obtain the pose estimation.
[0083] When the motion features are optimized and screened by the previous module, the visual odometry calculation method enters the last stage, i.e. the prediction pose stage. Pose estimation can be performed through the pose fitting module. The RCNN network model is still used in this module, and the RCNN network model is as shown in Figure 8 . After the calculation of the previous module, the two-layer LSTM network and the two-layer fully connected network are used to estimate the pose of the screened motion features. Since the description method of the camera relative motion of the algorithm is Euler angle, the model finally obtains a 6-dimensional vector as the output result of the pose estimation. At this time, the two-layer LSTM network in the model contains a 1024-parameter hidden layer, and each layer uses a Relu nonlinear activation function. The two linked fully connected layers gradually map the 1024-dimensional feature vector to 128-dimensional and 6-dimensional spaces.
[0084] The technical scheme of the embodiment of the application fully utilizes the optical flow information of the pixels in the input image sequence, extracts motion features on this basis, optimizes and screens the features through the motion consistency module, and finally completes the relative pose estimation task in a local range. The problems of poor generalization and difficulty in extracting motion consistency features in the current deep learning-based visual odometry calculation method are solved. Compared with the image sequence processing method of other deep visual odometry, the application fully considers the geometric constraint relationship of the camera motion between the input image sequence, and preferentially extracts the optical flow data of the input image sequence, and then performs the next step of pose estimation. Through the analysis of the geometric motion data, the application further screens the high-dimensional motion features of the camera through the motion consistency constraint module, and then fits the relative pose estimation of the camera.
[0085] In order to solve the problems existing in the visual odometer based on deep learning at present, including poor generalization, difficulty in extracting motion consistency features and the like, the application provides a visual odometer, which fully utilizes the optical flow information of pixels in an input image sequence, extracts motion features on the basis, optimizes and selects the features through a motion consistency module, and finally completes the estimation task of relative pose in a local range.
[0086] Figure 9 A structure schematic diagram of a pose estimation device based on a visual odometer is provided for the embodiment of the application. Figure 9 As shown in the figure, the device comprises:
[0087] An image processing module 910 is configured to acquire a to-be-processed image collected by a camera and extract optical flow data of the to-be-processed image based on an optical flow method.
[0088] A feature extraction module 920 is configured to extract features of the to-be-processed image through a multi-head attention mechanism and a feature extraction module according to the optical flow data, so as to obtain to-be-optimized motion features corresponding to the to-be-processed image.
[0089] A feature optimization module 930 is configured to optimize the to-be-optimized features according to a motion consistency constraint, so as to obtain target motion features corresponding to the to-be-processed image.
[0090] A pose estimation module 940 is configured to perform pose fitting based on the target motion features, so as to obtain pose estimation corresponding to the camera.
[0091] Optionally, the image processing module 910 comprises:
[0092] An optical flow data extraction module is configured to obtain optical flow data based on processing of the to-be-processed image.
[0093] The optical flow data extraction module comprises a feature pyramid extractor, a Warping layer and a cost volume.
[0094] Optionally, the formula for extracting the optical flow data is as follows:
[0095]
[0096] wherein I t-1 and I t represent continuous images collected by the camera at adjacent time points t-1 and t, is the optical flow data calculated by the function C, H and W represent the channel number, length and width of the optical flow data, wherein the channel number C=2 represents the motion status of the pixels in the horizontal and vertical coordinate directions.
[0097] Optionally, the feature extraction module 820 comprises a spatial feature extraction submodule, a temporal feature extraction submodule, and an attention module.
[0098] The spatial feature extraction submodule is configured to input the optical flow data and the to-be-processed image into the spatial feature extraction submodule to obtain spatial features of the to-be-processed image.
[0099] The temporal feature extraction submodule is configured to input the spatial features into the temporal feature extraction submodule to obtain temporal features of the to-be-processed image.
[0100] The attention mechanism module is configured to process the to-be-processed image through a multi-head attention mechanism to obtain multi-head attention mechanism features corresponding to the to-be-processed image.
[0101] The fusion module is configured to fuse the spatial features, the temporal features, and the multi-head attention mechanism features to obtain to-be-optimized motion features.
[0102] Optionally, the feature optimization module 930 is configured to:
[0103] Optimize the to-be-optimized features based on the learning parameters corresponding to the motion consistency constraint and the hidden state stored in the LSTM unit to obtain target motion features corresponding to the to-be-processed image.
[0104] Optionally, the pose estimation module 940 is specifically configured to:
[0105] Perform pose fitting on the target motion features based on a two-layer LSTM network and a two-layer fully connected network to obtain pose estimation.
[0106] The pose estimation device based on the visual odometry provided in the embodiments of the present application can execute the pose estimation method based on the visual odometry provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.
[0107] Figure 10 A structural schematic diagram of an electronic device for implementing the pose estimation method based on the visual odometry in the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown in the present document, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or claimed herein.
[0108] As Figure 10As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0109] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.
[0110] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the visual odometry based pose estimation method.
[0111] In some embodiments, the visual odometry based pose estimation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the visual odometry based pose estimation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the visual odometry based pose estimation method by any other appropriate means, such as by means of firmware.
[0112] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
[0113] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.
[0114] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0115] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
[0116] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0117] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.
[0118] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.
[0119] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.
Claims
1. A pose estimation method based on visual odometry, characterized in that: include: Obtaining an image to be processed captured by a camera, and extracting optical flow data of the image to be processed based on an optical flow method; According to the optical flow data, feature extraction is performed on the image to be processed through a multi-head attention mechanism and a feature extraction module to obtain motion features to be optimized corresponding to the image to be processed; Optimizing the feature to be optimized according to the motion consistency constraint to obtain the target motion feature corresponding to the image to be processed; Perform pose fitting based on the target operation characteristics to obtain a pose estimate corresponding to the camera.
2. The method according to claim 1, characterized in that The extracting and obtaining the optical flow data of the image to be processed based on the optical flow method includes: Processing the image to be processed based on the optical flow data extraction module to obtain the optical flow data; The optical flow data extraction module includes a feature pyramid extractor, a warping layer and a cost volume.
3. The method according to claim 2, characterized in that The formula for extracting the optical flow data is: Among them, I t-1 and I t represents the continuous images captured by the camera at adjacent times t-1 and t, is a function Calculated optical flow data; C, H, and W represent the number of channels, length, and width of the optical flow data, where the number of channels C = 2 represents the movement of pixels in the horizontal and vertical coordinate directions.
4. The method according to claim 1, wherein The feature extraction module includes a spatial feature extraction submodule and a temporal feature extraction submodule. The feature extraction of the image to be processed is performed through the multi-head attention mechanism and the feature extraction module to obtain the motion features to be optimized corresponding to the image to be processed, including: Inputting the optical flow data and the image to be processed into the spatial feature extraction submodule to obtain the spatial features of the image to be processed; Inputting the spatial features into the temporal feature extraction submodule to obtain the temporal features of the image to be processed; Processing the image to be processed by a multi-head attention mechanism to obtain a multi-head attention mechanism feature corresponding to the image to be processed; The spatial features, the temporal features and the multi-head attention mechanism features are fused to obtain the motion features to be optimized.
5. The method according to claim 1, wherein Optimizing the feature to be optimized according to the motion consistency constraint to obtain the target motion feature corresponding to the image to be processed includes: The feature to be optimized is optimized based on the learning parameters corresponding to the motion consistency constraint and the hidden state stored in the LSTM unit to obtain the target motion feature corresponding to the image to be processed.
6. The method according to claim 1, characterized in that The performing pose fitting based on the target operation characteristics to obtain a pose estimate corresponding to the camera includes: The pose estimation is obtained by performing pose fitting on the target operation features based on a two-layer LSTM network and a two-layer fully connected network.
7. A pose estimation device based on visual odometry, characterized in that: include: An image processing module is used to obtain an image to be processed captured by a camera and extract optical flow data of the image to be processed based on an optical flow method; A feature extraction module is used to extract features of the image to be processed based on the optical flow data through a multi-head attention mechanism and a feature extraction module to obtain motion features to be optimized corresponding to the image to be processed; A feature optimization module, configured to optimize the feature to be optimized according to a motion consistency constraint to obtain a target motion feature corresponding to the image to be processed; The pose estimation module is used to perform pose fitting based on the target operation characteristics to obtain a pose estimation corresponding to the camera.
8. The device according to claim 7, characterized in that The image processing module includes: An optical flow data extraction module, configured to obtain the optical flow data based on processing the image to be processed; The optical flow data extraction module includes a feature pyramid extractor, a warping layer and a cost volume.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the pose estimation method based on visual odometry according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the pose estimation method based on visual odometry according to any one of claims 1 to 6 when executed.