Deep learning camera pose estimation method, system and device and storage medium

Abstract

The invention relates to the technical field of image processing, and relates to a deep learning camera pose estimation method comprising the following steps: obtaining an image shot by a camera; extracting scene features of the image; extracting and integrating channel information, spatial information and regional attention information of the scene features by using a preset attention mechanism to obtain enhanced scene features; and determining the six-degree-of-freedom camera pose of the image according to the enhanced scene features. The invention further provides a deep learning camera pose estimation system and device and a storage medium. According to the invention, the feature extraction capability and adaptability of the deep learning model used for executing the deep learning camera pose estimation method can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep learning camera pose estimation method, system, device and storage medium. Background Art

[0002] Camera pose estimation refers to the technology of accurately estimating the six-degree-of-freedom pose of a camera in a world coordinate system in a known environment. Camera pose estimation has important applications in robotics and autonomous driving, and can significantly improve the accuracy of positioning and mapping. With the rapid development of deep learning, using deep learning to optimize camera pose estimation algorithms has become a current research hotspot. Currently, camera pose estimation algorithms such as PoseNet, Deep Visual Odometry (DeepVO for short) or Visual Localization Network++ (VLocNet++ for short) are usually used to process the input image to obtain the camera pose information.

[0003] However, existing camera pose estimation methods still have the following defects when dealing with large-scale outdoor scenes:

[0004] (1) Poor feature extraction capability: Existing camera pose estimation models rely on limited feature representation during training. Outdoor scenes are usually complex and prone to dynamic changes. When faced with unprecedented complex outdoor scenes, the model may not be able to capture enough discriminative features, resulting in a decrease in the accuracy of pose estimation.

[0005] (ii) Weak generalization ability: Existing deep learning-based camera pose estimation models often perform well on specific datasets, but their performance degrades significantly when applied to new scenes with a different distribution from the training data.

[0006] (III) Poor real-time performance: In large-scale outdoor scenes, due to the need to process a large number of pixels and features, inference time and memory usage remain a challenge on resource-constrained devices, resulting in poor real-time performance of the camera pose estimation model. Summary of the invention

[0007] In order to improve the feature extraction capability and adaptability of the model, a deep learning camera pose estimation method, system, device and storage medium are provided.

[0008] In order to achieve the above-mentioned objective of the present invention, the present invention provides a deep learning camera pose estimation method.

[0009] The present invention provides a deep learning camera pose estimation method, comprising:

[0010] Get the image captured by the camera;

[0011] Extract scene features of images;

[0012] The preset attention mechanism is used to extract and integrate the channel information, spatial information and regional attention information of the scene features to obtain enhanced scene features;

[0013] Determine the six-degree-of-freedom camera pose of the image based on the enhanced scene features.

[0014] Optionally, the extracting and integrating the channel information, spatial information and regional attention information of the scene features by using a preset attention mechanism to obtain enhanced scene features includes:

[0015] The channel attention mechanism is used to process scene features to obtain channel attention features, which are used to represent the channel information of the image.

[0016] Fusing the channel attention feature and the scene feature to obtain the first enhanced feature;

[0017] The first enhanced feature is processed by using a spatial attention mechanism to obtain a spatial attention feature, where the spatial attention feature is used to represent the spatial information of the image;

[0018] The spatial attention feature and the scene feature are integrated to obtain the second enhanced feature;

[0019] The key information attention module is used to process scene features to obtain key information attention features, which are used to characterize the attention degree of the image area.

[0020] The scene feature, the second enhanced feature and the key information attention feature are integrated to obtain the enhanced scene feature.

[0021] Optionally, the using a channel attention mechanism to process scene features to obtain channel attention features includes:

[0022] The channel-level statistical information in the feature map of the original feature is captured through DPP pooling and global maximum pooling, and the channel attention feature is obtained according to the channel-level statistical information;

[0023] The calculation formula of channel attention feature is:

[0024]

[0025] Among them, F represents the input scene feature, MLP represents multi-layer perceptron, DpPool represents DPP pooling operation, MaxPool represents maximum pooling operation, σ represents Sigmoid activation function, and is a set of weight parameters for MLP, C is the number of channels, r is the dimensionality reduction ratio, Represents the feature map of DPP pooling, Represents the feature map after maximum pooling, M c (F) represents the first enhanced feature output after being processed by the channel attention module.

[0026] Optionally, the using a spatial attention mechanism to process the first enhanced feature to obtain a spatial attention feature includes:

[0027] Performing global average pooling and global maximum pooling operations on the first enhanced feature to generate two two-dimensional feature maps, and concatenating and transforming the two feature maps to obtain the spatial attention feature;

[0028] The calculation formula of spatial attention feature is:

[0029]

[0030] Where F' represents the first enhanced feature map of the input, σ improve Represents the improved Sigmoid imporve Activation function, Represents the OctConv convolution operation with a convolution kernel size of 7x7, AvgPool(F') represents the features after average pooling on the channel axis, and MaxPool(F') represents the features after maximum pooling on the channel axis;

[0031] Improved Sigmoid imporve The activation function is

[0032]

[0033] Among them, x represents the input Sigmoid imporve The eigenvalue of the activation function, k is Sigmoid imporve The coefficients of the activation function.

[0034] Optionally, the scene features are processed using a key information attention module to obtain key information attention features, including:

[0035] Perform a deep separable convolution operation on the scene features to obtain the global information and contextual information of the scene features;

[0036] Perform deformable convolution operations on scene features to obtain attention information of the region in the image;

[0037] The key information attention features are determined based on the global and contextual information of the scene features and the attention information of the region in the image.

[0038] Optionally, a multi-level feature extraction and fusion network is used to process the image to obtain multi-level enhanced scene features, and the six-degree-of-freedom camera pose of the image is determined based on the multi-level enhanced scene features.

[0039] Optionally, the multi-level feature extraction fusion network includes:

[0040] A plurality of sequentially connected feature processing modules, wherein the feature processing modules include sequentially connected feature extraction modules and feature enhancement modules,

[0041] Among them, the feature extraction module is used to extract the features of the input image to obtain a scene feature map, and the feature enhancement module is used to extract and integrate the channel information, spatial information and regional attention information of the scene feature map according to a preset attention mechanism to obtain an enhanced scene feature map; specifically, the feature enhancement module first uses the channel attention mechanism to process the input to obtain the channel attention features for characterizing the channel information of the image, and fuses the channel attention features and the input features to obtain the first enhanced features of the input features; then uses the spatial attention mechanism to process the first enhanced features of the input features to obtain the spatial attention features for characterizing the spatial information of the image; fuses the spatial attention features and the scene features to obtain the second enhanced features of the input features; finally, uses the key information attention module to process the input features to obtain the key information attention features for characterizing the attention of the image area; fuses the input features, the second enhanced features of the input features and the key information attention features of the input features to obtain the enhanced scene features of the input features.

[0042] The channel attention mechanism, spatial attention mechanism and key information attention mechanism used by the feature enhancement module to enhance the input features are the same as those in step S3 of this embodiment and will not be described again here for the sake of brevity.

[0043] The feature fusion module is used to fuse the enhanced scene feature maps output by multiple feature enhancement modules to obtain multi-level enhanced scene features.

[0044] In order to solve the above problems, the present invention also provides a deep learning camera pose estimation system, the system comprising:

[0045] An acquisition module is used to acquire images taken by a camera;

[0046] An extraction module, used to extract scene features of an image;

[0047] An enhancement module is used to extract and integrate the channel information, spatial information and regional attention information of scene features using a preset attention mechanism to obtain enhanced scene features;

[0048] A generation module is used to determine the six-degree-of-freedom camera pose of the image based on the enhanced scene features.

[0049] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0050] at least one processor; and,

[0051] a memory communicatively connected to the at least one processor; wherein,

[0052] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the deep learning camera pose estimation method described above.

[0053] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned deep learning camera pose estimation method.

[0054] In summary, this application includes the following beneficial technical effects:

[0055] The attention mechanism can dynamically adjust the focus according to the different contents of the input image, and consider channel, space and saliency information when processing features, so that the deep learning model used to perform the deep learning camera pose estimation method in this application can pay more attention to the important parts of the image; different images may have different feature distributions and important areas. Through the processing of the preset attention mechanism, the deep learning model used to perform the deep learning camera pose estimation method can adaptively learn these differences and adjust the feature extraction and fusion strategies accordingly, so that the improved network has better adaptability and generalization ability when facing various different data sets and tasks, and the network can better extract and utilize key features, thereby maintaining good performance;

[0056] In addition, the multi-level feature fusion network integrates features at different levels, so that the deep learning model used to execute the deep learning camera pose estimation method can better adapt to different types of images and task requirements, improve the feature extraction ability and accuracy of the deep learning model used to execute the deep learning camera pose estimation method, thereby improving the accuracy of the camera pose estimation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a process flow of a deep learning camera pose estimation method provided by an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of a preset attention processing module provided in one embodiment of the present invention;

[0059] Figure 3 A flowchart of using a preset attention mechanism to process scene information to obtain enhanced scene information provided by an embodiment of the present invention;

[0060] Figure 4 A flow chart of processing scene features by a channel attention module provided in one embodiment of the present invention;

[0061] Figure 5 A flowchart of processing a first enhanced feature by a spatial attention module diagram provided in one embodiment of the present invention;

[0062] Figure 6 A flowchart of the processing of scene features by a key information attention module diagram provided by an embodiment of the present invention;

[0063] Figure 7 A flowchart of image processing by a multi-level feature extraction and fusion network provided in one embodiment of the present invention;

[0064] Figure 8 A schematic diagram of the structure of an electronic device for implementing the deep learning camera pose estimation method provided by one embodiment of the present invention.

[0065] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0066] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0068] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0069] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0070] Reference Figure 1 FIG. 1 is a flow chart of a deep learning camera pose estimation method provided by an embodiment of the present invention. In this embodiment, the deep learning camera pose estimation method includes:

[0071] S1. Obtain an image captured by a camera.

[0072] S2. Extract scene features of the image.

[0073] The input image is processed using a deep learning model (such as a fully convolutional network (FCN)) to extract the scene features of the image. The scene features include not only texture information but also depth information, thus providing a basis for the subsequent camera 3D coordinate prediction.

[0074] S3. Use the preset attention mechanism to extract and integrate the channel information, spatial information and regional attention information of the scene features to obtain enhanced scene features.

[0075] Reference Figure 2 and Figure 3 , the preset attention mechanism is used to extract and integrate the channel information, spatial information and regional attention information of the scene features to obtain enhanced scene features, including:

[0076] S31. Use the channel attention mechanism to process scene features and obtain channel attention features.

[0077] Channel attention features are used to represent the channel information of an image; see Figure 4 , the scene features are processed through the channel attention module. The channel attention module captures the channel-level statistics in the feature map of the original features through DPP pooling and global maximum pooling, and then uses a shared fully connected layer (or convolutional layer) to learn the weight of each channel, and finally generates a channel attention map through the sigmoid activation function. This attention map is multiplied with the original feature map to enhance the features of important channels and obtain the channel attention features;

[0078] The calculation formula of channel attention feature is:

[0079]

[0080] Among them, F represents the input scene feature, MLP represents multi-layer perceptron, DpPool represents DPP pooling operation, MaxPool represents maximum pooling operation, σ represents Sigmoid activation function, and is a set of weight parameters of MLP, C represents the number of channels, r is the dimensionality reduction ratio, Represents the feature map of DPP pooling, Represents the feature map after maximum pooling, M c (F) represents the first enhanced feature output after being processed by the channel attention module.

[0081] The full name of DPP pooling is Detail-Preserving Pooling. Detail-Preserving Pooling (DPP) is a new pooling method used for image processing in deep learning, which aims to reduce the amount of calculation while retaining the key details of the image to the maximum extent. The commonly used average pooling adopts the same average strategy for all regions, while DPP pooling can dynamically adjust the pooling strategy according to the content of the input feature map, which is an adaptive pooling method; in addition, the internal parameters of DPP pooling can be learned through back propagation, which enables DPP pooling to be optimized for specific tasks, while the parameters of average pooling are fixed and cannot be learned; DPP pooling allows downsizing to focus on important structural details, and the learnable parameters control the amount of detail preservation, which can achieve better results.

[0082] S32: Fuse the channel attention feature and the scene feature to obtain the first enhanced feature.

[0083] Specifically, the first enhanced feature can be obtained by performing element-by-element multiplication of the elements in the channel attention feature and the elements in the scene feature.

[0084] S33. Use the spatial attention mechanism to process the first enhanced feature to obtain a spatial attention feature.

[0085] The spatial attention feature is used to represent the spatial information of the image. The spatial attention module is used to process the first enhanced feature and pay attention to the importance of different spatial positions in the feature map; refer to Figure 5 , the spatial attention module performs global average pooling and global maximum pooling operations on the first enhanced feature to generate two two-dimensional feature maps, then concatenates the two feature maps, performs feature transformation through a convolution layer, and finally generates a spatial attention map through a sigmoid function. This attention map will also be multiplied with the original feature map to enhance the features of important positions and obtain the spatial attention feature;

[0086] The calculation formula of spatial attention feature is:

[0087]

[0088] Where F' represents the first enhanced feature map of the input, σ impove Represents the improved Sigmoid imporve Activation function, Represents the OctConv convolution operation with a convolution kernel size of 7x7, AvgPool(F') represents the features after average pooling on the channel axis, and MaxPool(F') represents the features after maximum pooling on the channel axis;

[0089] Improved Sigmoid imporve The activation function is

[0090]

[0091] Among them, x represents the characteristic value of the input data, x can be any real number; k is Sigmoid imporve The coefficient of the activation function, in this embodiment, k is a hyperparameter used to control the slope of the function. It affects the growth rate of the function, that is, the speed at which the function value approaches 1 when the input value x increases. The larger the value of k, the faster the function grows in the positive direction; the smaller the value of k, the slower the function grows in the positive direction.

[0092] This application takes computational efficiency into consideration in the design of the channel attention module and the spatial attention module. By introducing DPP pooling and advanced OctConv convolution operations, performance is improved while maintaining low additional computational costs. The network model extraction capability for executing deep learning camera pose estimation methods is improved while the model has better real-time performance.

[0093] S34, integrating the spatial attention feature and the scene feature to obtain the second enhanced feature.

[0094] Specifically, the second enhanced feature can be obtained by performing element-by-element multiplication of the elements in the spatial attention feature and the elements in the scene feature.

[0095] S35. Use the key information attention module to process scene features to obtain key information attention features.

[0096] Key information attention features are used to characterize the attention of image regions; refer to Figure 2 , the extraction of channel information and spatial information of scene features is a simple serial structure, that is, the input features are first processed by the channel attention module, and then by the spatial attention module; however, there is a major flaw in this, in the serial structure, the output of the channel attention module will be sent to the spatial attention module. In this process, because the channel attention module may have reconstructed the feature map to a certain extent, some information useful to the spatial attention module may be lost; by extracting the key information attention features, the problem of the original feature information loss can be solved.

[0097] The key information attention features are extracted as follows: performing a deep separable convolution operation on the scene features to obtain the global information and contextual information of the scene features; performing a deformable convolution operation on the scene features to obtain the attention information of the area in the image; and determining the key information attention features based on the global information, contextual information and attention information.

[0098] Reference Figure 2 In the actual implementation process, the key information attention features of the image can be extracted by adding a parallel branch (key information attention module) to the processing link of the channel attention module and the spatial attention module; wherein the channel attention module and the spatial attention module are connected in series, and the processing links corresponding to the channel attention module and the spatial attention module are in parallel with the processing links corresponding to the key information attention module.

[0099] Reference Figure 6 In this embodiment, the key information attention module is a lightweight plug-and-play DAS, and the full name of DAS in English is A Deformable Attention to Capture Salient Information in CNNs); DAS is a new type of attention mechanism, which can help convolutional neural networks (CNNs) focus on key information in the image more effectively. DAS can solve the problem of original CBAM feature information loss by combining depthwise separable convolution (Depthwise Separable Convolution) and deformable convolution (Deformable Convolution), where depthwise separable convolution is used to efficiently represent the global context, while deformable convolution is used to increase attention to related image areas. The computational complexity of DAS is only O(n), so adding a parallel branch of the DAS attention module on the basis of the channel attention module and the spatial attention module will not significantly increase the computational burden, but it can solve the problem of original scene feature information loss. And by adding parallel branches (key information attention modules) to the serial channel attention module and spatial attention module, channel, spatial and saliency information can be considered simultaneously in the same network layer without waiting for one module to be completed before processing another module, thus improving processing efficiency.

[0100] The following formula explains how the preset attention mechanism applies channel attention and spatial attention in sequence to enhance the input feature map.

[0101]

[0102] Among them, F is the input scene feature map, M c is the channel attention map, M sis the spatial attention map, M DAS is the feature map processed by the key information attention module, Represents element-by-element multiplication. During the multiplication process, the attention values ​​are broadcast (copied) accordingly: the channel attention values ​​are broadcast along the spatial dimension, and vice versa. F′ is the channel attention feature obtained after processing by the channel attention module, F″ is the spatial attention feature after processing by the channel attention module and the spatial attention module, and F″′ is the enhanced scene feature output after the second enhanced feature is fused with the key information attention feature.

[0103] S36, integrating the scene feature, the second enhanced feature and the key information attention feature to obtain an enhanced scene feature.

[0104] The vector element corresponding to the scene feature, the vector element corresponding to the second enhanced feature, and the vector element corresponding to the key information attention feature are multiplied to obtain the final enhanced scene feature.

[0105] The attention mechanism allows the deep learning model used to perform the deep learning camera pose estimation method to focus on key areas in the image, which helps to improve the robustness of the model in the face of noise, occlusion and complex backgrounds; the attention mechanism can dynamically adjust the focus of attention according to the different contents of the input image. Different images may have different feature distributions and important areas. Through the attention mechanism, the network can adaptively learn these differences and adjust the feature extraction and fusion strategies accordingly. This makes the improved network have better adaptability and generalization ability when facing a variety of different data sets and tasks. For example, when dealing with complex situations such as different lighting conditions, angle changes, object occlusion, etc., the network can better extract and utilize key features, thereby maintaining good performance.

[0106] S4. Determine the six-degree-of-freedom camera pose of the image according to the enhanced scene features.

[0107] The camera pose is usually represented by a 3x3 rotation matrix and a 3x1 translation vector. The rotation matrix is ​​used to describe the rotation of the camera, and the translation vector is used to describe the translation of the camera. The six-degree-of-freedom camera pose refers to the position and posture of the camera in three-dimensional space, including three translation degrees of freedom and three rotation degrees of freedom; the three translation degrees of freedom represent the movement distance of the camera on the X, Y, and Z axes, and the three rotation degrees of freedom represent the rotation angle of the camera around the X, Y, and Z axes.

[0108] Based on the extracted enhanced scene features, the 3D scene coordinates corresponding to each pixel in the image are predicted. Specifically, the above steps are usually implemented through one or more convolutional layers, and finally a feature map with the same resolution as the input image is output, where each pixel value represents the coordinate information in the corresponding 3D space; using the predicted 3D coordinates, multiple possible camera pose hypotheses are generated through random sampling or optimization methods; finally, each pose hypothesis is scored through a differentiable scoring function, the most likely pose hypothesis is selected, and the pose estimate may be refined through further optimization steps to obtain the final six-degree-of-freedom camera pose.

[0109] In this embodiment, after establishing the mapping relationship between dense 2D image pixels and 3D scene coordinates, the camera pose estimation model can use the RANSAC and PnP algorithms to calculate the camera pose; RANSAC is an iterative algorithm for estimating the parameters of a mathematical model from a data set containing outliers. The RANSAC algorithm identifies inliers and outliers through multiple random samplings, and finally selects the mathematical model with the most inliers as the optimal mathematical model; PnP is Perspective-n-Point, and the PnP algorithm is used to solve the camera pose problem, that is, through the known camera internal parameters and the corresponding relationship between several feature points, the pose of the camera in the world coordinate system is calculated to obtain the final six-degree-of-freedom camera pose.

[0110] In the preferred implementation of this embodiment, a multi-level feature extraction and fusion network is used to process the image to obtain multi-level enhanced scene features, and the six-degree-of-freedom camera pose of the image is determined based on the multi-level enhanced scene features. Through multi-level feature fusion, the network can combine features at different levels, such as low-level detail information and high-level semantic information, to obtain richer feature representation.

[0111] Reference Figure 7 ,The multi-level feature extraction and fusion network includes: a number of sequentially connected feature ,processing modules, and a feature fusion module;

[0112] A plurality of sequentially connected feature processing modules, the feature processing modules comprising a feature extraction module and a feature enhancement module connected in sequence, wherein the feature extraction module is used to extract features of an input image to obtain a scene feature map, and the feature enhancement module is used to extract and integrate channel information, spatial information, and regional attention information of the scene feature map according to a preset attention mechanism to obtain an enhanced scene feature map;

[0113] The enhanced scene feature map output by the first set of feature processing modules is used as the input image of the second set of feature processing modules; the feature fusion module is used to fuse the enhanced scene feature maps output by multiple feature enhancement modules to obtain multi-level enhanced scene features.

[0114] Figure 7 The CRB module in is the feature extraction module, which is used to extract features. Figure 7 The CBAM_DAS in the figure is a feature enhancement module, which is used to enhance features. The features enhanced by the attention module are then fused with the original scene features processed by the feature extraction module to better regress the scene coordinates. Conv represents a simple convolution operation, which is mainly used to adjust the number of output channels. GAP is a global average pooling, which is used to compress the previously extracted features to prevent neural network redundancy. The last connection layer plays the role of mapping the learned feature representation to the sample's label space. In other words, it integrates the features together. Finally, RANSAC+PnP is used to parse the extracted image features, solve the coordinates of the 2D image pixels in the 3D scene, and obtain the final required six-degree-of-freedom camera pose.

[0115] Figure 7 The camera pose recognition network shown takes an RGB image as input with a default image resolution of 640×480 and generates a feature map with a resolution of 80×60. Figure 7 The camera pose recognition network shown can integrate low-level edge structure information and high-level spatial detail features to complete scene parsing. In addition, CBAM_DAS enables the network to focus more on effective information by learning feature maps at different levels. Finally, the network regresses and predicts dense three-dimensional scene coordinates through the powerful feature representation of the former. It should be explained that the difference between low-level information and high-level information lies in the number of CRB operations they have undergone. Note that the essence of the CRB module is actually a convolution operation Conv plus a Relu activation function. In convolutional neural networks, features are extracted through convolution operations. The fewer convolution operations the original input features undergo, the closer the features it outputs are to the original features. At this time, they are called low-level features, which mainly contain edge structure information. The more convolution operations it undergoes, the higher the output features are called high-level features, which generally contain more spatial detail features. Of course, whether it is low-level or high-level needs to be defined in combination with the specific network architecture, and there is no rigid standard. With Figure 7 As an example, Figure 7 The features output by the CRB1 module in is the low-level edge structure information; Figure 7 The features output by the CRB3 module in are high-level spatial detail features. Figure 7 The features output by the CRB2 module in are between the two and are intermediate-level features.

[0116] The multi-level feature fusion network can better adapt to different types of images and task requirements by integrating features at different levels, and improve the processing efficiency and adaptability of deep learning models used to perform deep learning camera pose estimation methods.

[0117] Based on the same inventive concept, an embodiment of the present invention provides a deep learning camera pose estimation system.

[0118] The deep learning camera pose estimation system described in the present invention can be loaded in an electronic device. According to the functions implemented, the deep learning camera pose estimation system includes an acquisition module, an extraction module, an enhancement module and a generation module. The acquisition module can acquire images taken by a camera; the extraction module can extract scene features of the image; the enhancement module can extract and integrate channel information, spatial information and regional attention information of the scene features using a preset attention mechanism to obtain enhanced scene features; the generation module can determine the six-degree-of-freedom camera pose of the image based on the enhanced scene features.

[0119] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0120] The various variations and specific examples of the deep learning camera pose estimation method provided in the above embodiments are also applicable to the deep learning camera pose estimation system of the present embodiment. Through the above detailed description of the deep learning camera pose estimation method, those skilled in the art can clearly understand the implementation method of the deep learning camera pose estimation system in the present embodiment. For the sake of brevity, it will not be described in detail here.

[0121] The present application also discloses an electronic device, such as Figure 8 , which is a schematic diagram of the structure of an electronic device for a deep learning camera pose estimation method provided by an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 connected to the at least one processor for communication, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a deep learning camera pose estimation method program.

[0122] Among them, the processor 10 can be composed of an integrated circuit in some embodiments, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executes a deep learning camera pose estimation method, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0123] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the deep learning camera pose estimation method program, but also can be used to temporarily store data that has been output or is to be output.

[0124] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0125] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0126] Figure 8 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0127] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0128] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0129] The present application embodiment provides a computer-readable storage medium, for example, including: any entity or device, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM) that can carry the computer program code. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the deep learning camera pose estimation method of the above embodiment.

[0130] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0131] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A deep learning camera pose estimation method, characterized in that: Get the image captured by the camera; Extract scene features of images; The preset attention mechanism is used to extract and integrate the channel information, spatial information and regional attention information of the scene features to obtain enhanced scene features; Determine the six-degree-of-freedom camera pose of the image based on the enhanced scene features.

2. The deep learning camera pose estimation method according to claim 1, characterized in that: The method uses a preset attention mechanism to extract and integrate channel information, spatial information, and regional attention information of scene features to obtain enhanced scene features, including: The channel attention mechanism is used to process scene features to obtain channel attention features, which are used to represent the channel information of the image. Fusing the channel attention feature and the scene feature to obtain the first enhanced feature; The first enhanced feature is processed by using a spatial attention mechanism to obtain a spatial attention feature, where the spatial attention feature is used to represent the spatial information of the image; The spatial attention feature and the scene feature are integrated to obtain the second enhanced feature; The key information attention module is used to process scene features to obtain key information attention features, which are used to characterize the attention degree of the image area. The scene feature, the second enhanced feature and the key information attention feature are integrated to obtain the enhanced scene feature.

3. The deep learning camera pose estimation method according to claim 2, characterized in that: The channel attention mechanism is used to process scene features to obtain channel attention features, including: The channel-level statistical information in the feature map of the original feature is captured through DPP pooling and global maximum pooling, and the channel attention feature is obtained according to the channel-level statistical information; The calculation formula of channel attention feature is: Among them, F represents the input scene features, MLP represents multi-layer perceptron, DpPool represents DPP pooling operation, MaxPool represents maximum pooling operation, σ represents Sigmoid activation function, and is a set of weight parameters of MLP, C represents the number of channels, r is the dimensionality reduction ratio, Represents the feature map of DPP pooling, Represents the feature map after maximum pooling, M c (F) represents the first enhanced feature output after being processed by the channel attention module.

4. The deep learning camera pose estimation method according to claim 2, characterized in that: The method of processing the first enhanced feature using the spatial attention mechanism to obtain the spatial attention feature includes: Performing global average pooling and global maximum pooling operations on the first enhanced feature to generate two two-dimensional feature maps, and concatenating and transforming the two feature maps to obtain the spatial attention feature; The calculation formula of spatial attention feature is: Where F' represents the first enhanced feature map of the input, σ improve Represents the improved Sigmoid imporve Activation function, Represents the OctConv convolution operation with a convolution kernel size of 7x7, AvgPool(F') represents the features after average pooling on the channel axis, and MaxPool(F') represents the features after maximum pooling on the channel axis; Improved Sigmoid imporve The activation function is Among them, x represents the input Sigmoid imporve The eigenvalue of the activation function, k is Sigmoid imporve The coefficients of the activation function.

5. The deep learning camera pose estimation method according to claim 2, characterized in that: The key information attention module is used to process scene features to obtain key information attention features, including: Perform a deep separable convolution operation on the scene features to obtain the global information and contextual information of the scene features; Perform deformable convolution operations on scene features to obtain attention information of the region in the image; The key information attention features are determined based on the global and contextual information of the scene features and the attention information of the region in the image.

6. The deep learning camera pose estimation method according to any one of claims 1 to 5, characterized in that: The image is processed using a multi-level feature extraction and fusion network to obtain multi-level enhanced scene features, and the six-degree-of-freedom camera pose of the image is determined based on the multi-level enhanced scene features.

7. The deep learning camera pose estimation method according to claim 6, characterized in that: The multi-level feature extraction fusion network includes: A plurality of sequentially connected feature processing modules, the feature processing modules comprising a feature extraction module and a feature enhancement module connected in sequence, wherein the feature extraction module is used to extract features of an input image to obtain a scene feature map, and the feature enhancement module is used to extract and integrate channel information, spatial information, and regional attention information of the scene feature map according to a preset attention mechanism to obtain an enhanced scene feature map; The feature fusion module is used to fuse the enhanced scene feature maps output by multiple feature enhancement modules to obtain multi-level enhanced scene features.

8. A deep learning camera pose estimation system, used to implement the deep learning camera pose estimation method according to any one of claims 1 to 7, characterized in that: include: An acquisition module is used to acquire images taken by a camera; An extraction module, used to extract scene features of an image; An enhancement module is used to extract and integrate the channel information, spatial information and regional attention information of scene features using a preset attention mechanism to obtain enhanced scene features; A generation module is used to determine the six-degree-of-freedom camera pose of the image based on the enhanced scene features.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the deep learning camera pose estimation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed by a processor, the deep learning camera pose estimation method according to any one of claims 1 to 7 is implemented.

Images