Pose determination method and device in visual mapping, equipment and medium

Abstract

The invention discloses a pose determination method and device in visual mapping, equipment and a medium. The method comprises the following steps: acquiring a previous feature map of a previous frame of image and a current feature map of a current frame of image; determining a primary matching result of elements in the previous feature map and the current feature map through a self-attention mechanism and a cross attention mechanism; performing cascade optimization on the primary matching result to obtain a target matching result of elements in the previous frame image and the current frame image; and resolving the pose of the current frame based on the target matching result. According to the embodiment of the invention, a stable matching result can be realized in a conventional scene and an extreme scene.

Description

Technical Field

[0001] The present invention relates to the technical field of posture determination in visual mapping, and in particular to a posture determination method, device, equipment and medium in visual mapping. Background Art

[0002] Monocular vision mapping technology is widely used in fields such as autonomous driving, robotics, and virtual reality. Among them, how to achieve effective element matching between monocular image data and establish stable data association relationships between pixels; and how to establish visual maps through the above association relationships are one of the key issues that need to be addressed in this technical field.

[0003] Regarding the former, one of the current mainstream technical solutions is to use a feature extraction algorithm to select representative pixels as feature points based on changes in grayscale, texture, etc. in the image, and generate a corresponding description vector for each feature point to calculate the similarity of the feature points. The data association problem in the mapping process is then solved by matching the feature points and descriptors detected in the two frames of images.

[0004] Data association based on feature points can achieve accurate mapping in conventional scenes with rich image features. However, when dealing with degraded scenes such as missing textures and single elements (such as in tunnels), the mapping accuracy will be reduced due to the lack of stable feature points. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for determining posture in visual mapping, so as to achieve stable matching results in both conventional and extreme scenarios.

[0006] According to one aspect of the present invention, a method for determining a pose in a visual map is provided, comprising:

[0007] Obtain the previous feature map of the previous frame image and the current feature map of the current frame image;

[0008] Determine the primary matching result of the elements in the previous feature map and the current feature map through a self-attention mechanism and a cross-attention mechanism;

[0009] Performing cascade optimization on the primary matching results to obtain target matching results of elements in the previous frame image and the current frame image;

[0010] The current frame pose is solved based on the target matching result.

[0011] According to another aspect of the present invention, a posture determination device in visual mapping is provided, comprising:

[0012] A feature extraction module is used to obtain a previous feature map of a previous frame image and a current feature map of a current frame image;

[0013] A matching determination module, used to determine the primary matching results of elements in the previous feature map and the current feature map through a self-attention mechanism and a cross-attention mechanism;

[0014] A matching optimization module, used for performing cascade optimization on the primary matching results to obtain target matching results of elements in the previous frame image and the current frame image;

[0015] The pose solving module is used to solve the pose of the current frame based on the target matching result.

[0016] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method for determining a pose in visual mapping according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for determining a pose in visual mapping as described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for determining a pose in visual mapping described in any embodiment of the present invention when executed.

[0019] The embodiment of the present invention does not rely on feature extraction. Given any two frames of visual images, the matching results of dense image elements are output through an end-to-end matching model. Stable matching results can be achieved in both conventional and extreme scenarios, thereby improving the accuracy of subsequent pose determination and visual mapping.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1A is a flow chart of a method for determining a pose in a visual map according to an embodiment of the present invention;

[0023] Figure 1B is a schematic diagram of the structure of an end-to-end matching model provided according to an embodiment of the present invention;

[0024] Figure 1C is a schematic diagram of a posture determination process provided according to an embodiment of the present invention;

[0025] Figure 2A is a flowchart of a method for determining a pose in visual mapping provided according to another embodiment of the present invention;

[0026] Figure 2B is a schematic diagram of a key frame screening process provided according to another embodiment of the present invention;

[0027] Figure 2C is a schematic diagram of a visual mapping process provided according to another embodiment of the present invention;

[0028] Figure 3 is a structural schematic diagram of a method for determining a posture in visual mapping provided according to another embodiment of the present invention;

[0029] Figure 4 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Figure 1A A flowchart of a method for determining a pose in visual mapping provided by an embodiment of the present invention. This embodiment can be applied to the current frame image captured by a monocular camera, and the matching result of the elements in the image is obtained by matching it with the previous frame image data through an end-to-end matching model, and pose analysis and visual mapping are performed accordingly. The method can be executed by a pose determination device in visual mapping, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device with corresponding data processing capabilities, such as a visual mapping system. Figure 1A As shown, the method includes:

[0033] S110, obtaining a previous feature map of a previous frame image and a current feature map of a current frame image.

[0034] S120, determining the primary matching results of the elements in the previous feature map and the current feature map through a self-attention mechanism and a cross-attention mechanism.

[0035] S130, performing cascade optimization on the primary matching results to obtain target matching results of elements in the previous frame image and the current frame image.

[0036] S140, calculating the current frame pose based on the target matching result.

[0037] The elements are pedestrians, traffic lights, guardrails and other common objects on the road captured by the monocular camera on the vehicle during driving. The matching result is mathematically described as a set r of several two-dimensional point pairs, that is, Among them, each element in the set represents the position of the matching result on the i-th frame and the j-th frame image respectively.

[0038] Specifically, the system first performs an initialization step to determine the initial pose of the map coordinate system and generate local map data. The system is initialized only once each time it runs, and only needs to be initialized again when a failure occurs in the subsequent process or when it is re-run. After the initialization is completed, the system's work is mainly divided into three independent threads, namely the tracking thread, the optimization thread, and the loop thread. In the tracking thread, the system needs to fuse the matching results of the elements in the current frame with the previous frame and the matching results of the current frame with the elements in the local map, so as to solve the pose of the current frame camera in the map coordinate system, that is, the current frame pose.

[0039] The matching results of the elements in the current frame and the previous frame are determined by an end-to-end matching model, which consists of four different modules. Figure 1B shown.

[0040] Module 1 Figure 1B As shown in the upper left part, this part extracts features from the input image through the basic network. Since the convolutional neural network can fully extract the local features of the image, resnet50 can be used as the basic network. Considering that image matching is sensitive to the original position of the element, the first pooling operation in resnet50 is cancelled, and only the convolution parameters of the first three stages are retained. The three layers of the basic network output feature maps of 1 / 2*1 / 2, 1 / 4*1 / 4, and 1 / 8*1 / 8 original image sizes respectively. Based on the above improvements, the previous frame image and the current frame image are input into the basic network, and the two feature maps of 1 / 8*1 / 8 original image size output by the third layer are determined as the previous feature map and the current feature map.

[0041] Module 2 Figure 1B As shown in the lower left part, this module is an improved Transformer module, and the input of this module is the previous feature map and the current feature map. In the Transformer module, the self-attention mechanism and the cross-attention mechanism are used to extract and determine the correlation information of the elements in the previous feature map and the current feature map, and the extracted correlation information is fused to the feature map and output.

[0042] Module 3 Figure 1B As shown in the lower right part, the two feature maps output by the improved Transformer module contain rich correlation information. The module will further extract and analyze the correlation information in the feature map to obtain the primary matching results of the elements in the previous feature map and the current feature map.

[0043] Module 4 Figure 1BAs shown in the upper right part, since the feature map has downsampled the element image, the primary matching result cannot be directly output as the final matching result. It also needs to be cascaded optimized. In this module, the primary matching result output by the third module is fused with the first-layer feature map output by the basic network, and the cascade optimization of the matching relationship is performed to obtain the target matching result, which is output as the final matching result.

[0044] Based on the target matching results, valid matching elements are determined. When the matched elements have been associated with the map data in the previous frame, the association results are synchronized to the current frame. For unassociated elements (such as pedestrians that were not in the previous frame), the present invention projects the local map data to the image coordinate system according to the camera imaging principle, and at the same time, the local map data is associated with the data of the current frame pixel points through the nearest neighbor matching strategy, and the posture is optimized by constructing the reprojection error.

[0045] The reprojection error reprojects the spatial point onto the two-dimensional image plane through the imaging principle of the camera. By calculating the distance between the position after projection and the position of the original pixel, and taking this distance as the optimization target, the camera pose and the spatial point position are optimized. Taking an element in a frame of image as an example, the above error can be mathematically calculated by formula (1):

[0046] e rep =||Ω(R cT p m -R T p c )-f|| (1)

[0047] Where RC and pc represent the posture and position of the camera in the world coordinate system, pm is the map point in the three-dimensional space, f is the element associated with the map point in the image coordinate system, and the function Ω is the camera projection model.

[0048] If the error defined by formula (1) is optimized by nonlinear optimization method, the initial estimation of the camera pose in the current frame can be achieved. Figure 1C As shown, further, the pixel points in the end-to-end matching result that are not associated with the map data can be triangulated according to the initial pose to obtain new local map elements. Finally, the present invention updates the newly generated map elements to the local map data, and reconstructs the augmented reprojection error to obtain the final pose result of the camera in the current frame.

[0049] The embodiment of the present invention does not rely on feature extraction. Given any two frames of visual images, the matching results of dense image elements are output through an end-to-end matching model. Stable matching results can be achieved in both conventional and extreme scenarios, thereby improving the accuracy of subsequent pose determination and visual mapping.

[0050] Figure 2A This is a flowchart of a method for determining a pose in a visual map provided by another embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. Figure 2A As shown, the method includes:

[0051] S210, obtaining a previous feature map of a previous frame image and a current feature map of a current frame image.

[0052] S220, extracting the internal correlation of the elements in the previous feature map and the current feature map through a self-attention mechanism; extracting the external correlation of the elements in the previous feature map and the current feature map through a cross-attention mechanism.

[0053] S230. Determine a primary matching result of elements in the previous feature map and the current feature map according to the internal correlation and the external correlation.

[0054] Among them, internal correlation refers to the correlation between elements within a feature map, and external correlation refers to the correlation between elements in one feature map and elements in another feature map.

[0055] For details, please refer to Figure 1B , input the previous feature map and the current feature map into the Transformer module, and obtain the previous feature map and the current feature map output by the Transformer module that integrate the internal correlation information and the external correlation information. Input it into the Transformer module again, and obtain the previous feature map and the current feature map output by the Transformer module that integrate the internal correlation information and the external correlation information. In this way, repeat the preset ((N times, N>1) times, so that the Transformer module repeatedly extracts the correlation information to obtain sufficient correlation information and improve the richness of the correlation information.

[0056] After completing N times of correlation extraction through the Transformer module, the previous feature map and the current feature map output by the Transformer module for the last time are input into module 3, and the correlation information is sorted and summarized by module 3 to obtain the primary matching result.

[0057] Based on the above embodiment, optionally, before extracting the internal correlation of elements in the previous feature map and the current feature map by the self-attention mechanism, the method further includes:

[0058] Performing downsampling processing on the previous feature map and the current feature map;

[0059] Correspondingly, after marking the internal correlation and the external correlation on the previous feature map and the current feature map, the method further includes:

[0060] Upsampling is performed on the previous feature map and the current feature map.

[0061] For details, please refer to Figure 1B In module 2, in order to improve the execution efficiency of the algorithm, the local similarity characteristics in the image matching problem are considered at the same time. Before using the attention mechanism to extract the correlation information, the input feature map is first downsampled using the deep convolution technology, and the correlation information is extracted after the downsampling is completed. Correspondingly, after the extraction is completed, the feature map is processed by upsampling to keep its size consistent with the input, and then the feature map is output after upsampling.

[0062] Based on the above embodiment, optionally, determining the primary matching result of the elements in the previous feature map and the current feature map according to the internal correlation and the external correlation includes:

[0063] The internal correlation and the external correlation are marked on the previous feature map and the current feature map; the correlation between the elements in the previous feature map and the elements in the current feature map is extracted again through the cross-attention mechanism to obtain a first correlation score matrix; based on a preset classification regression algorithm, the primary matching results of the elements in the previous feature map and the current feature map are determined according to the first correlation score matrix.

[0064] The first correlation score matrix records the correlation scores of each element in the previous feature map and each element in the current feature map. The first correlation score matrix is ​​mathematically an N*N matrix, where N represents the number of input features, and the element at the (i, j)th position of the matrix represents the correlation between the i-th element in the previous feature map and the j-th element in the current feature map.

[0065] For details, please refer to Figure 1B Module 3 uses the cross attention mechanism (it is worth noting that the cross attention in this module does not perform downsampling) to process the previous feature map and the current feature map of the input respectively. The obtained results are processed by the dual-normalized exponential function (dual-softmax) to obtain the first correlation score matrix. Module 3 uses a classification regression algorithm (such as the mutual-nearest-neighbor algorithm, MNN) to process the first correlation score matrix and obtain the primary matching results of the elements in the above two feature maps.

[0066] S240, mapping the primary matching result to the initial previous feature map of the previous frame image and the initial current feature map of the current frame image through upsampling;

[0067] S250. Extract the correlation between the elements in the previous cropped feature map and the elements in the current cropped feature map through a cross-attention mechanism to obtain a second correlation score matrix; based on a preset classification regression algorithm, determine the target matching results of the elements in the previous frame image and the current frame image according to the second correlation score matrix.

[0068] Among them, the correlation score matrix records the correlation between each element in the initial previous feature map and each element in the initial current feature map; the size of the initial previous feature map is larger than the previous feature map, and the size of the initial current feature map is larger than the current feature map.

[0069] Specifically, refer to Figure 1B , in theory, the primary matching result is related to the data in a certain area of ​​the first-layer feature map. Therefore, in module 4, the primary matching result is first mapped to the first-layer feature map through 4 times upsampling, and a window of 8*8 size is selected with this point as the center to crop the first-layer features. The feature map obtained by the cropping operation is subjected to the same cross attention and fusion processing as the third module, so that the secondary optimization matching can be achieved, and then the final matching relationship is determined; specifically: the input initial previous feature map and initial current feature map are processed separately through the cross attention mechanism. And the obtained results are processed by a double normalized exponential function (dual-softmax), and the second correlation score matrix is ​​output. This module will use the classification regression algorithm to process the second correlation score matrix to obtain the matching results of the elements in the above two feature maps. Since the sizes of the initial previous feature map and the initial current feature map are close to the original image, the matching results of the elements in these two feature maps can more realistically reflect the matching results of the elements in the original image. Therefore, it is determined as the target matching result of the elements in the previous frame image and the current frame image.

[0070] S260, determining a key frame in the sliding window according to the pose of each frame in the sliding window; when a valid loop is retrieved, correcting the pose of the key frame and the map data within the loop range by a preset optimization algorithm.

[0071] Specifically, the optimization thread implements key frame screening based on the distance the camera moves and the change in parallax. This operation can remove redundant data in the frame image sequence and select representative image frames as key frames. Subsequent optimization thread work is based on key frames. The workflow of the optimization thread is as follows: Figure 2BAs shown, given several frames of images and the position and posture of the camera corresponding to each frame of image (solved by tracking thread), three thresholds are set, namely, the two-frame position threshold, the angle threshold and the successful matching point threshold, respectively. When the distance and angle between the current frame and the previous key frame exceed the first two thresholds, or the number of valid matches between the current frame and the previous frame is less than the third threshold, the current frame is set as the key frame.

[0072] The optimization thread is implemented based on keyframes. In order to improve the execution efficiency of the algorithm, the optimization in the invention is implemented within the sliding window, that is, only the keyframes within the sliding window of a fixed size are optimized. Similar to the tracking thread, the error construction is also implemented through the reprojection error. The difference is that the variables to be optimized are the poses of multiple keyframe cameras, and the optimization of local map data elements is newly added.

[0073] In order to eliminate the accumulated errors in the tracking thread and the optimization thread in solving the camera pose, such as Figure 2C As shown in the figure, consistent with the current mainstream visual mapping solutions, the present invention also adds a loop detection thread to the system, which is implemented through the bag-of-words model. Please refer to ORB-SLAM2. This part is not a core technology and will not be described in detail. Figure 2C The solution of sliding window optimization is the same as that of the present invention, except that the optimization range is the entire loop interval, and the optimization content is still the position and posture of the frame image. The nonlinear optimization problem solved in the present invention can be implemented based on the Levenberg-Marquardt (LM) algorithm.

[0074] The embodiment of the present invention improves the optimization efficiency of posture data by selecting key frames according to posture data.

[0075] Figure 3 A structural diagram of a posture determination device in visual mapping provided by another embodiment of the present invention. Figure 3 As shown, the device comprises:

[0076] The feature extraction module 310 is used to obtain the previous feature map of the previous frame image and the current feature map of the current frame image;

[0077] A matching determination module 320, configured to determine a primary matching result of elements in the previous feature map and the current feature map through a self-attention mechanism and a cross-attention mechanism;

[0078] A matching optimization module 330, configured to perform cascade optimization on the primary matching results to obtain target matching results of elements in the previous frame image and the current frame image;

[0079] The pose solving module 340 is used to solve the pose of the current frame based on the target matching result.

[0080] The apparatus for determining posture in visual mapping provided in an embodiment of the present invention can execute the method for determining posture in visual mapping provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0081] Optionally, the matching determination module 320 includes:

[0082] A first correlation extraction unit, configured to extract internal correlations of elements in the previous feature map and the current feature map through a self-attention mechanism;

[0083] A second correlation extraction unit, configured to extract external correlations between elements in the previous feature map and the current feature map through a cross attention mechanism;

[0084] A matching determination unit is used to determine a primary matching result of elements in the previous feature map and the current feature map according to the internal correlation and the external correlation.

[0085] Optionally, the matching determination unit is specifically used to: mark the internal correlation and the external correlation on the previous feature map and the current feature map; extract the correlation between the elements in the previous feature map and the elements in the current feature map again through the cross-attention mechanism to obtain a first correlation score matrix; record the correlation score between each element in the previous feature map and each element in the current feature map in the first correlation score matrix; based on a preset classification and regression algorithm, determine the primary matching results of the elements in the previous feature map and the current feature map according to the first correlation score matrix.

[0086] Optionally, the matching determination module 320 further includes:

[0087] A preprocessing unit, used to downsample the previous feature map and the current feature map

[0088] A post-processing unit is used to upsample the previous feature map and the current feature map.

[0089] Optionally, the matching optimization module 330 includes:

[0090] A matching mapping unit, used for mapping the primary matching result to an initial previous feature map of the previous frame image and an initial current feature map of the current frame image by upsampling; the size of the initial previous feature map is larger than the previous feature map, and the size of the initial current feature map is larger than the current feature map;

[0091] A feature map clipping unit, used for clipping the initial previous feature map and the initial current feature map to obtain a previous clipped feature map and a current clipped feature map respectively;

[0092] A score calculation unit, used for extracting the correlation between the elements in the previous cropped feature map and the elements in the current cropped feature map through a cross attention mechanism to obtain a second correlation score matrix; the correlation score matrix records the correlation between each element in the initial previous feature map and each element in the initial current feature map;

[0093] A matching optimization unit is used to determine the target matching results of the elements in the previous frame image and the current frame image according to the second correlation score matrix based on a preset classification regression algorithm.

[0094] Optionally, the device further comprises:

[0095] A key frame determination module, used to determine the key frame in the sliding window according to the pose of each frame in the sliding window;

[0096] The loop detection module is used to correct the key frame pose and map data within the loop range through a preset optimization algorithm when a valid loop is retrieved.

[0097] The further described apparatus for determining a pose in visual mapping may also execute the method for determining a pose in visual mapping provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0098] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0099] like Figure 4As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0100] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0101] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 41 executes the various methods and processes described above, such as the pose determination method in visual mapping.

[0102] In some embodiments, the pose determination method in visual mapping may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the pose determination method in the visual mapping described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the pose determination method in the visual mapping by any other appropriate means (e.g., by means of firmware).

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining pose in visual mapping, characterized in that: The method comprises: Obtain the previous feature map of the previous frame image and the current feature map of the current frame image; Determine the primary matching result of the elements in the previous feature map and the current feature map through a self-attention mechanism and a cross-attention mechanism; Performing cascade optimization on the primary matching results to obtain target matching results of elements in the previous frame image and the current frame image; The current frame pose is solved based on the target matching result.

2. The method according to claim 1, characterized in that: The determining of the primary matching result of the elements in the previous feature map and the current feature map by the self-attention mechanism and the cross-attention mechanism includes: Extracting the internal correlation of elements in the previous feature map and the current feature map through a self-attention mechanism; Extracting external correlations between elements in the previous feature map and the current feature map through a cross-attention mechanism; A primary matching result of elements in the previous feature map and the current feature map is determined according to the internal correlation and the external correlation.

3. The method according to claim 2, characterized in that The determining, according to the internal correlation and the external correlation, a primary matching result of elements in the previous feature map and the current feature map comprises: Marking the internal correlation and the external correlation on the previous feature map and the current feature map; The correlation between the elements in the previous feature map and the elements in the current feature map is extracted again through the cross attention mechanism to obtain a first correlation score matrix; the first correlation score matrix records the correlation score between each element in the previous feature map and each element in the current feature map; Based on a preset classification regression algorithm, a primary matching result of the elements in the previous feature map and the current feature map is determined according to the first correlation score matrix.

4. The method according to claim 3, characterized in that: Before extracting the internal correlation of elements in the previous feature map and the current feature map by the self-attention mechanism, the method further includes: Performing downsampling processing on the previous feature map and the current feature map; Correspondingly, after marking the internal correlation and the external correlation on the previous feature map and the current feature map, the method further includes: Upsampling is performed on the previous feature map and the current feature map.

5. The method according to claim 1, characterized in that The step of performing cascade optimization on the primary matching results to obtain target matching results of elements in the previous frame image and the current frame image includes: Mapping the primary matching result to an initial previous feature map of the previous frame image and an initial current feature map of the current frame image by upsampling; the size of the initial previous feature map is larger than the previous feature map, and the size of the initial current feature map is larger than the current feature map; The initial previous feature map and the initial current feature map are respectively cropped to obtain a previous cropped feature map and a current cropped feature map; The correlation between the elements in the previous cropped feature map and the elements in the current cropped feature map is extracted through a cross attention mechanism to obtain a second correlation score matrix; the correlation score matrix records the correlation between each element in the initial previous feature map and each element in the initial current feature map; Based on a preset classification regression algorithm, target matching results of elements in the previous frame image and the current frame image are determined according to the second correlation score matrix.

6. The method according to claim 1, characterized in that After calculating the position and posture of the camera in the current frame based on the target matching result, the method further includes: Determine a key frame in the sliding window according to the pose of each frame in the sliding window; When a valid loop is retrieved, the key frame pose and map data within the loop range are corrected through a preset optimization algorithm.

7. A posture determination device in visual mapping, characterized in that: The device comprises: A feature extraction module is used to obtain a previous feature map of a previous frame image and a current feature map of a current frame image; A matching determination module, used to determine the primary matching results of elements in the previous feature map and the current feature map through a self-attention mechanism and a cross-attention mechanism; A matching optimization module, used for performing cascade optimization on the primary matching results to obtain target matching results of elements in the previous frame image and the current frame image; The pose solving module is used to solve the pose of the current frame based on the target matching result.

8. The device according to claim 7, characterized in that The matching determination module comprises: A first correlation extraction unit, configured to extract internal correlations of elements in the previous feature map and the current feature map through a self-attention mechanism; A second correlation extraction unit, configured to extract external correlations between elements in the previous feature map and the current feature map through a cross attention mechanism; A matching determination unit is used to determine a primary matching result of elements in the previous feature map and the current feature map according to the internal correlation and the external correlation.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the pose determination method in visual mapping according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the pose determination method in visual mapping according to any one of claims 1 to 6 when executed.

Images