Video scene mapping and positioning method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video scene mapping and positioning method, comprising: a scene map establishment process based on iterative pose optimization; a video spatio-temporal feature aggregation process; a spatio-temporal region matching process based on saliency weighting; a multi-video geographical location recognition and scene merging process; a video geographical location positioning process from coarse to fine. The present disclosure also provides a video scene mapping and positioning device, an electronic device and a readable storage medium.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of autonomous driving, mobile robot navigation, simultaneous localization and mapping, video geo-location recognition and positioning, and the like, and particularly relates to a video scene mapping and positioning method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Due to the emergence of application scenarios such as autonomous memory parking, intelligent logistics trolley, restaurant intelligent robot meal delivery, and unmanned aerial vehicle autonomous cruising, how to recognize a scene that has been reached is very important.

[0003] These application scenarios often pre-plan a correct motion path and establish a scene map when performing a task for the first time (such as parking a car in a home parking space), and then the intelligent robot or autonomous driving vehicle autonomously performs the task according to the current observed scene, perceives which position in the scene map it is in, and then autonomously follows the pre-planned path or autonomously avoids obstacles according to the scene map.

[0004] The following two points are crucial for the above-mentioned application scenarios. First, to establish a sufficiently accurate scene map, i.e., a video mapping process. Second, a positioning link, i.e., a process of perceiving which position in the scene map it is in, mainly a geo-location recognition and positioning.

[0005] In the above-mentioned application scenarios, the surrounding environment of the scene may have changed greatly during the mapping and positioning processes, for example, the mapping route is long and takes a long time to loop back; the mapping is performed in the morning and the positioning is performed at night; the mapping is performed on a sunny day and the autonomous navigation is performed on a rainy, foggy, or snowy day, or even there may be a cross-season situation, resulting in a great difference in the appearance of the observed scene. In addition, the scenes of these applications are often very complex, for example, the interference of dynamic objects such as pedestrians and vehicles during mapping and positioning, thereby further increasing the difference in the appearance of the observed scene and even these dynamic objects may cause local occlusion of the scene; at the same time, some open scenes or repeated appearance of similar textures are also a big challenge, such as an open parking lot, similar design styles of different garages, repeated appearance of almost identical lamp poles and fences on the road, etc. In these scenarios, video mapping and positioning are very difficult, which limits the accuracy of geo-location recognition and positioning.

[0006] In some scenarios, it is impossible to determine which position in the scene map is currently in by relying on a single picture. However, when humans encounter such a scene, they usually look around, even walk a short distance first, and then make a judgment based on the overall scene information. In recent years, some scholars have begun to explore the extension of single-frame visual repositioning to video sequences from practical applications, sacrificing the real-time performance of some algorithms to improve the accuracy of repositioning by using the spatiotemporal fusion of multiple frames. Especially in the industrial field, such as automatic parking and other application scenarios, the human driver usually drives the vehicle to the entrance of the garage and then starts the autonomous parking system, so at this time the vehicle's geographical position at the moment when the system is started can be located according to the previous driving route. The core difficulty of visual repositioning based on video sequences is how to fuse the spatiotemporal features for geographical position retrieval.

[0007] For example, in the paper "Sarlin P E, Unagar A, Larsson M, et al. Back to the Feature: Learning Robust Camera Localization from Pixels to Pose [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 3247-3257.", a feature map image alignment method is used for pose calculation, and a confidence image is generated to weight the feature point projection error, so that the accuracy of pose calculation is greatly higher than that of the traditional feature point matching method. However, the scene retrieval stage of this paper still uses the NetVLAD method, and the retrieval accuracy is low.

[0008] For another example, in the paper "Spencer J, Bowden R, Hadfield S. Same features, different day: Weakly supervised feature learning for seasonal invariance [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 6459-6468.", the similarity between two images is defined by one-to-one matching of feature points, thereby constructing a training framework for a continuous multi-frame image three-element loss, and obtaining feature points that are robust to seasonal changes. However, the method of this paper can only be used for feature point matching and pose calculation between two images, and the paper uses the reference image pose with the highest similarity as the initial pose.

[0009] For another example in patent document CN110609917A, the saliency clues of each image layer are fused into a saliency map by using a graph model, a layered saliency algorithm is used to extract the saliency region of an image, a VGG19 is used to extract features of the saliency region, and the saliency region features are stored in an image database for image retrieval. The way of synthesizing a saliency image and extracting region features one by one by using a VGG19 has a high computational complexity.

[0010] For another example in patent document US10304237B2, a relocalization system is constructed by map point projection, a k-d tree is trained by all feature points in a map for scene retrieval, an initial 6DOF pose is calculated by minimizing the re-projection error of feature points between two frames, and a sliding window is constructed for local pose iterative optimization to obtain an accurate relocalization pose. The relocalization system has the advantages of strong generalization and low computational complexity, but mainly relies on low-dimensional image features such as image feature points, and has a high failure rate in relocalization when the external environment such as illumination and viewing angle changes.

[0011] In general, the disadvantages of the current technology include: low positioning accuracy after scene appearance changes, low retrieval recall rate, not robust to illumination and viewing angle changes; there is no way to deal with the influence of foreground occlusion, and the model accuracy is greatly disturbed; when the model is trained, it cannot correctly determine whether it is in the same geographical location when the images are only partially related; there is only geographical location recognition for single-frame images, and the technology based on video sequences is not mature. SUMMARY

[0012] The present disclosure provides a video scene mapping and positioning method, device, electronic equipment and storage medium.

[0013] According to one aspect of the present disclosure, a video scene mapping and positioning method is provided, comprising:

[0014] S100, constructing an image pyramid for a current observation image of a current observation video sequence, obtaining an initial pose, a confidence pyramid and a dense feature map of the current observation image based on the image pyramid, and performing iterative pose optimization in a scene map based on the initial pose, the confidence pyramid and the dense feature map of the current observation image to obtain an optimized pose, so as to obtain a scene map including the optimized pose of the current observation image;

[0015] S200, fusing and processing the depth features of all frame images of the current observation video sequence to obtain a global descriptor of video spatio-temporal feature fusion of the current observation video sequence;

[0016] S300, performing video geolocation recognition using the global descriptor to obtain a plurality of initial candidate video sequences with the highest similarity of geographical location to the current observed video sequence from a pre-constructed video sequence database, and performing local region feature extraction on all frame images of the current observed video sequence to obtain region descriptors of a plurality of regions of each frame image;

[0017] S400, performing region matching based on region descriptors between each frame image of the current observed video sequence and each frame image of each initial candidate video sequence, obtaining a spatial consistency degree of each initial candidate video sequence with the current observed video sequence based on a matching result, and further obtaining a plurality of final candidate video sequences corresponding to the spatial consistency degree from the plurality of initial candidate video sequences;

[0018] S500, for each final candidate video sequence, obtaining a co-view image cluster of each frame image in a scene map to fuse each independent final candidate video sequence into a single scene, and obtaining a pose of the current observed image, i.e., a camera pose, based on a feature matching result of all 3D map points of each co-view image cluster in the scene map and feature points of the current observed image;

[0019] S600, based on the pose of the current observed image, performing iterative pose optimization from low-resolution images to high-resolution images based on an image pyramid to perform scene matching between the current observed image and each single scene, and obtaining a geographical location positioning result of the current observed image.

[0020] According to the method for video scene mapping and positioning of at least one embodiment of the present disclosure, in step S100, further comprising:

[0021] performing feature extraction on the image pyramid based on a convolutional neural network with an "encoder-decoder" structure to obtain a feature pyramid.

[0022] According to the method for video scene mapping and positioning of at least one embodiment of the present disclosure, in step S100, obtaining an initial pose of the current observed image based on the image pyramid comprises:

[0023] S1102, performing image geolocation retrieval in a pre-constructed video sequence database based on the image pyramid to obtain a plurality of candidate images with the highest similarity of geographical location to the current observed image;

[0024] S1104, obtaining an initial pose of the current observed image based on a co-view relationship of the plurality of candidate images.

[0025] According to the method for video scene mapping and positioning according to at least one embodiment of the present disclosure, in step S100, the confidence pyramid of the current observation image is obtained based on the image pyramid, including: confidence calculation is performed on the feature pyramid to obtain the confidence pyramid.

[0026] In the confidence pyramid, different layers represent the region confidence of different scales, and each layer corresponds to the saliency of the corresponding position of the image pyramid.

[0027] According to the method for video scene mapping and positioning according to at least one embodiment of the present disclosure, the confidence calculation on the feature pyramid includes: S1202, inputting the feature pyramid into an SK-Net network based on a self-attention mechanism; and S1204, inputting the output of the SK-Net network as input and performing convolution processing on the first convolution layer to obtain the confidence pyramid.

[0028] According to the method for video scene mapping and positioning according to at least one embodiment of the present disclosure, in step S100, the dense feature map of the current observation image is obtained based on the image pyramid, including:

[0029] The dense feature calculation is performed on the feature pyramid to obtain the dense feature map of the current observation image.

[0030] According to the method for video scene mapping and positioning according to at least one embodiment of the present disclosure, the dense feature calculation on the feature pyramid includes:

[0031] The feature pyramid is input into the second convolution layer for convolution processing to obtain the dense feature map of the current observation image.

[0032] According to the method for video scene mapping and positioning according to at least one embodiment of the present disclosure, in step S100, the confidence pyramid and the dense feature map are used to perform iterative pose optimization in the scene map based on the initial pose of the current observation image, and an optimized pose is obtained, including:

[0033] S1502, obtaining the 3D map points of each candidate image in the co-view image cluster having a co-view relationship in the scene map to form a 3D map point set;

[0034] S1504, projecting the 3D map points in the 3D map point set onto the image coordinate system of the current observation image to determine whether each 3D map point is located within the current observation image;

[0035] S1506, obtaining a visible map point set based on the 3D map points located within the current observation image;

[0036] S1508, convert the projection point of each map point in the set of visible map points in the current observation image into a dense feature coordinate in the dense feature map to obtain a corresponding dense feature of each map point in the set of visible map points;

[0037] S1510, based on the confidence pyramid corresponding to the current observation image, weight processing the dense feature corresponding to each map point in the set of visible map points and the multiple projection features of each map point on the current observation image to highlight the salient features of the current observation image, to align the overall image features of the current observation image and the candidate image;

[0038] S1512, taking the pose of the current observation image after the overall image feature alignment as an initial value, performing iterative pose optimization from low-resolution image to high-resolution image based on the image pyramid to obtain the optimized pose.

[0039] According to the video scene mapping and positioning method of at least one embodiment of the present disclosure, in step S1512, taking the pose of the current observation image after the overall image feature alignment as an initial value, performing iterative pose optimization from image features of low-resolution image to image features of high-resolution image based on the image pyramid, comprising:

[0040] sequentially performing the following iterative pose optimization process from image features of low-resolution image to image features of high-resolution image based on the image pyramid:

[0041] based on the confidence of the confidence pyramid corresponding to the current resolution image, weight processing the dense feature corresponding to each map point in the set of visible map points and the multiple projection features of each map point on the current resolution image to highlight the salient features of the current resolution image, to align the image features of the current resolution image and the candidate image.

[0042] According to the video scene mapping and positioning method of at least one embodiment of the present disclosure, in step S1512, performing the iterative pose optimization process based on the LM algorithm.

[0043] According to the video scene mapping and positioning method of at least one embodiment of the present disclosure, further comprising:

[0044] output the current observation image and its corresponding confidence image to obtain a visualization result, and output the scene map including the optimized pose of the current observation image to obtain a visualization result.

[0045] The method for video scene mapping and positioning according to at least one embodiment of the present disclosure, the pre-constructed video sequence database comprises a plurality of video sequences, and the video sequences are obtained based on the following method:

[0046] Key frame image extraction is performed on the acquired video stream to obtain a video sequence composed of key frame images, and then a plurality of video sequences are obtained.

[0047] The method for video scene mapping and positioning according to at least one embodiment of the present disclosure, the video sequence database comprises a plurality of video sequences, and the video sequences are obtained based on the following method:

[0048] Scene depth information extraction is performed on all key frame images of each video sequence to construct a 3D perspective space body based on key frame image scene depth information for each video sequence;

[0049] The degree of coincidence of the 3D perspective space body of each video sequence is judged to obtain the scene similarity between the corresponding scenes of each video sequence; and

[0050] The common view relationship of each key frame image is obtained based on the scene similarity between the corresponding scenes of each video sequence.

[0051] The method for video scene mapping and positioning according to at least one embodiment of the present disclosure, S200, the depth features of all frame images of the current observation video sequence are fused to obtain a video spatio-temporal feature fused global descriptor of the current observation video sequence, comprising:

[0052] S2102, feature extraction is performed on all frame images of the current observation video sequence to obtain corresponding feature maps;

[0053] S2104, the feature maps of all frame images are stacked in the channel dimension, and the features of each frame image are enhanced to perform time domain fusion on the features of each frame image, to obtain the feature maps of each frame image after time domain fusion;

[0054] S2106, the feature maps of each frame image after time domain fusion are spliced to obtain a video sequence feature map;

[0055] S2108, residual merging and spatio-temporal feature aggregation are performed on the video sequence feature map to generate the global descriptor.

[0056] The method for video scene mapping and positioning according to at least one embodiment of the present disclosure, the global descriptor is represented using a high-dimensional vector.

[0057] According to the video scene mapping and positioning method of at least one embodiment of the present disclosure, in step S300, local region feature extraction is performed on all frame images of the current observation video sequence to obtain region descriptors of multiple regions of each frame image, including:

[0058] S3102, each layer of the feature pyramid corresponding to the image pyramid of the frame image is subjected to feature enhancement and feature clustering processing to obtain multiple cluster centers;

[0059] S3104, each feature of the feature pyramid is subjected to weighted residual error calculation with each cluster center to obtain a residual error pyramid corresponding to the feature pyramid;

[0060] S3106, for each layer of the residual error pyramid, region extraction is performed using a sliding window of a preset size to obtain multiple region blocks of each layer of the residual error pyramid, and a residual integral graph of each region block is obtained;

[0061] S3108, based on the residual integral graph, region descriptors of multiple regions of each layer of the feature pyramid corresponding to the residual error pyramid are obtained.

[0062] According to the video scene mapping and positioning method of at least one embodiment of the present disclosure, in step S400, region matching based on region descriptors is performed on each frame image of the current observation video sequence and each frame image of each initial candidate video sequence, and based on the matching result, the spatial consistency degree of each initial candidate video sequence and the current observation video sequence is obtained, and then one or more than two final candidate video sequences corresponding to the spatial consistency degree are obtained from the multiple initial candidate video sequences, including:

[0063] S4102, the distance between all region descriptors of each frame image of the current video observation sequence and all region descriptors of other frame images is calculated to obtain a distance matrix, and a region matching set between different frame images is obtained based on the distance matrix;

[0064] S4104, the region descriptor of each frame image in the region matching set with the minimum average distance to all region descriptors of other frame images is taken as the representative region descriptor of each frame image to obtain the representative region descriptor of the current observation video sequence;

[0065] S4106, region matching is performed based on the representative region descriptor of each frame image of the current observation video sequence and the representative region descriptor of each frame image of each initial candidate video sequence to obtain a preliminary matching result;

[0066] S4108, performing weighted matching on the preliminary matching result based on the confidence pyramids of the frame images of the current observation video sequence and each of the initial candidate video sequences, to obtain a spatial consistency degree between each two frame images between the current observation video sequence and each of the initial candidate video sequences as the weighted matching result;

[0067] S4110, obtaining a spatial consistency degree between the current observation video sequence and each of the initial candidate video sequences based on the spatial consistency degrees between each two frame images between the current observation video sequence and each of the initial candidate video sequences, and further obtaining a plurality of final candidate video sequences corresponding to the spatial consistency degree from the plurality of initial candidate video sequences.

[0068] According to the method for video scene mapping and positioning according to at least one embodiment of the present disclosure, in step S500, for each of the final candidate video sequences, a co-view image cluster of each frame image in the scene map is obtained to fuse each independent final candidate video sequence into a separate scene, and a pose of a current observation image, i.e., a camera pose, is obtained based on feature matching results of all 3D map points of each co-view image cluster in the scene map and feature points of the current observation image, including:

[0069] S5102, for each of the final candidate video sequences, frame images having a co-view relationship in the scene map and a co-view ratio greater than or equal to a preset threshold are taken out one by one to obtain a co-view image cluster having a co-view relationship greater than or equal to a co-view ratio threshold with each frame image;

[0070] S5104, performing deduplication processing on images in each co-view image cluster to fuse each independent final candidate video sequence into a separate scene;

[0071] S5106, for each co-view image cluster after deduplication processing, images belonging to an initial candidate video sequence are screened, and a similarity between a global descriptor of the current observation video sequence and an initial candidate video sequence to which the screened image belongs is used as a score of the screened image, and a score of an image not belonging to an initial candidate video sequence is 0, and scores of images in each co-view image cluster are accumulated to serve as a score of each co-view image cluster;

[0072] S5108, based on an order of scores of each co-view image cluster from high to low, the following steps are sequentially performed: all 3D map points of the co-view image cluster are taken out from the scene map; feature points of a last frame of the current observation video sequence, i.e., a current observation image, are extracted to perform 2D-3D feature matching with the 3D map points; and a pose of the current observation image, i.e., a camera pose, is solved based on the feature matching result and serves as an initial pose of the co-view image cluster.

[0073] According to another aspect of the present disclosure, there is provided an apparatus for video scene mapping and positioning, comprising:

[0074] a scene map establishing module, configured to: construct an image pyramid for a current observed image of a current observed video sequence, obtain an initial pose, a confidence pyramid and a dense feature map of the current observed image based on the image pyramid, perform an iterative pose optimization in a scene map based on the initial pose, the confidence pyramid and the dense feature map of the current observed image and obtain an optimized pose, so as to obtain a scene map comprising the optimized pose of the current observed image;

[0075] a video spatio-temporal feature aggregation module, configured to: perform a fusion processing on depth features of all frame images of the current observed video sequence, so as to obtain a globally aggregated descriptor of the current observed video sequence;

[0076] a spatio-temporal region matching module, configured to: perform a video geographical location recognition using the globally aggregated descriptor, obtain a plurality of initial candidate video sequences with the highest geographical location similarity to the current observed video sequence from a pre-constructed video sequence database, perform a local region feature extraction on all frame images of the current observed video sequence, so as to obtain region descriptors of a plurality of regions of each frame image, perform a region matching based on the region descriptors between each frame image of the current observed video sequence and each frame image of each initial candidate video sequence, obtain a spatial consistency degree of each initial candidate video sequence and the current observed video sequence based on a matching result, and further obtain a plurality of final candidate video sequences corresponding to the spatial consistency degree from the plurality of initial candidate video sequences;

[0077] a multi-video geographical location recognition and scene merging module, configured to: for each final candidate video sequence, obtain a co-view image cluster in a scene map for each frame image, so as to fuse each independent final candidate video sequence into a single scene, and obtain a pose of a current observed image, i.e. a camera pose, based on a feature matching result between all 3D map points of each co-view image cluster in the scene map and feature points of the current observed image;

[0078] a video geographical location positioning module, configured to perform an iterative pose optimization from a low-resolution image to a high-resolution image based on an image pyramid based on the pose of the current observed image, so as to perform a scene matching between the current observed image and each single scene, and obtain a geographical location positioning result of the current observed image.

[0079] According to yet another aspect of the present disclosure, there is provided an electronic device comprising: a memory storing execution instructions; and a processor executing the execution instructions stored by the memory, so that the processor executes the method of video scene mapping and localization according to any one of the embodiments of the present disclosure.

[0080] According to yet another aspect of the present disclosure, there is provided a readable storage medium having stored therein execution instructions for implementing the method of video scene mapping and localization according to any one of the embodiments of the present disclosure when executed by a processor.

[0081] Most of the existing scene recognition and retrieval technologies are based on image global descriptors, such as NetVlad (Network Vector of Locally Aggregated Descriptors), SOLAR (Second-Order Loss and Attention for Image Retrieval), and the like. The image global descriptor-based scene retrieval has the advantages of fast retrieval speed and high storage efficiency, but it ignores the detailed information in the image, and the recall rate of scene retrieval is low. Therefore, some embodiments of the present disclosure fuse image global descriptors, region descriptors based on confidence maps, and point descriptors to construct a hierarchical scene retrieval architecture, which greatly improves the accuracy of scene retrieval. Most of the existing relocalization technologies are based on feature point matching, such as SuperPoint, HF-Net (Hierarchical Localization Network), and the like. The robustness of these technologies to changes in illumination and viewing angle is high, but the accuracy of pose calculation during relocalization is low due to the existence of multi-stage errors. Therefore, some embodiments of the present disclosure use the LM algorithm to directly align and optimize the image, which uniformly optimizes the errors and improves the accuracy during relocalization.

[0082] Some embodiments of the present disclosure fuse image global descriptors, region descriptors based on confidence maps, and point descriptors to construct a hierarchical scene retrieval architecture, which greatly improves the accuracy of scene retrieval. Some embodiments of the present disclosure calculate the initial pose according to the point matching results of multiple reference images in the scene retrieval stage, and simultaneously unify the scene retrieval and pose calculation into one network architecture, which greatly reduces the computational load and strengthens the coupling relationship between the two.

[0083] Some embodiments of the present disclosure construct an end-to-end relocalization architecture, which not only enables large-scale scene retrieval, but also iteratively optimizes the initial pose obtained in the scene retrieval stage, thereby greatly improving the accuracy of relocalization.

[0084] Some embodiments of the present disclosure directly obtain a saliency image using a Unet network, then generate saliency regions using connected component detection, and then directly obtain region features by mapping the residual map in the Netvlad layer. This region feature generation method is not only fast and direct, but also has high repeatability of region extraction. The present disclosure also integrates point matching, iterative pose optimization and other modules, so that end-to-end pose repositioning can be performed.

[0085] Some embodiments of the present disclosure consider high-scale semantic information such as saliency regions and image global descriptors, and use deep learning networks for image alignment and pose optimization, which has strong robustness to environmental appearance changes. BRIEF DESCRIPTION OF DRAWINGS

[0086] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description, explain the principles of the present disclosure, in which:

[0087] Figure 1 is a different generation method of existing video sequence descriptors.

[0088] Figure 2 is an existing SeqNet video geolocation retrieval architecture based on time series convolution.

[0089] Figure 3 is a camera observable region modeling (view cone model) used by the present disclosure.

[0090] Figure 4 is a flowchart of a method of video scene mapping and positioning of an embodiment of the present disclosure.

[0091] Figure 5 is a hierarchical feature alignment and projection optimization geolocation positioning (HAPOR) network module diagram of the present disclosure.

[0092] Figure 6 is a current observation image and its corresponding dense feature map and confidence map of an embodiment of the present disclosure.

[0093] Figure 7 is a running visualization result of a video scene mapping and positioning system designed by the present disclosure.

[0094] Figure 8 is a flowchart of an embodiment of the present disclosure for performing iterative pose optimization in the scene map based on the initial pose of the current observation image, the confidence pyramid and the dense feature map, and obtaining the optimized pose.

[0095] Figure 9is a flowchart of obtaining a global descriptor of a video spatio-temporal feature fusion of a current observation video sequence according to an embodiment of the present disclosure.

[0096] Figure 10 is a structure diagram of a Temporal VLAD network designed according to the present disclosure.

[0097] Figure 11 is a flowchart of obtaining a region descriptor of a plurality of regions of each frame image according to an embodiment of the present disclosure.

[0098] Figure 12 is a flowchart of obtaining a plurality of final candidate video sequences corresponding to a degree of spatial consistency from a plurality of initial candidate video sequences according to an embodiment of the present disclosure.

[0099] Figure 13 is a structure diagram of an image region feature extraction network designed according to the present disclosure.

[0100] Figure 14 is an auxiliary diagram of an integral graph optimization algorithm according to an embodiment of the present disclosure.

[0101] Figure 15 is a flowchart of a spatio-temporal region matching algorithm based on saliency weighting according to an embodiment of the present disclosure.

[0102] Figure 16 is a flowchart of multi-video geographical location recognition and scene merging according to an embodiment of the present disclosure.

[0103] Figure 17 is a structure diagram of a device for video scene mapping and positioning using a hardware implementation of a processing system according to an embodiment of the present disclosure.

[0104] Reference sign explanation

[0105] 1000 device for video scene mapping and positioning

[0106] 1002 scene map establishing module

[0107] 1004 video spatio-temporal feature aggregation module

[0108] 1006 spatio-temporal region matching module

[0109] 1008 multi-video geographical location recognition and scene merging module

[0110] 1010 video geographical location positioning module

[0111] 1100 bus

[0112] 1200 processor

[0113] 1300 memory

[0114] 1400 other circuitry. DETAILED DESCRIPTION

[0115] The present disclosure will be further described in details with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are merely exemplary for the purpose of explanation and are not limiting to the present disclosure. In addition, it should be noted that only parts related to the present disclosure are shown in the drawings for the purpose of description.

[0116] It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0117] Unless otherwise specified, the exemplary embodiments / instances shown will be understood to provide exemplary features of various details that can implement the technical concepts of the present disclosure in practice. Therefore, unless otherwise specified, the features of various embodiments / instances can be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of the present disclosure.

[0118] In the drawings, cross-hatching and / or shading are generally used to indicate that a portion of one feature can be located structurally close to another, even though not directly visible. As such, unless stated, the presence or absence of cross-hatching or shading is not a requirement or indication of any particular material, material property, size, scale, commonality of the illustrated components, or any other characteristic, attribute, property, etc. of the components. In addition, in the drawings, the size and relative sizes of components can be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be practiced differently, a specific process sequence can be performed in a different order than described. For example, two consecutively described processes can be performed at substantially the same time or in the opposite order to that described. In addition, like reference numerals indicate like components.

[0119] When a component is referred to as being “on” or “over” another component, “connected to” or “coupled to” another component, it can be directly on, directly connected to, or directly coupled to the other component, or intervening components can be present. However, when a component is referred to as being “directly on” or “directly connected to” or “directly coupled to” another component, there are no intervening components present. For this reason, the term “connected” can refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0120] The terminology used by the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms "including," "includes," "having," "has," "a" or "an," or the like are used in the detailed description and / or claims, such terms are intended to be inclusive (i.e., to say that at least the stated feature is present), unless the context clearly indicates otherwise. It will also be noted that, as used herein, the terms "substantially," "approximately," and other like terms are used as terms of approximation and not as terms of degree, unless the context clearly indicates otherwise. Thus, at the very least, such terms are termed in a manner that will be understood by those of ordinary skill in the art, and even if exact amounts or conditions are called out, such terms are to be interpreted as indicating approximations.

[0121] Video sequence geo-location recognition and positioning status quo:

[0122] Video sequence geo-location retrieval is a new research direction in recent years, and currently mainly relies on aggregating the descriptors of each image in the video sequence to generate a descriptor for the video sequence. For example, Delta first generates a global descriptor for each image in the video sequence, and then obtains a weighted descriptor for the entire sequence using smoothing.

[0123] Figure 1 are different ways of generating existing video sequence descriptors. As shown in Figure 1 (a) respectively through splicing of single image global descriptors to constitute a global descriptor of a video sequence, (b) through a fully connected layer to fuse them, (c) using a long short-term memory network (LSTM) to summarize the information of single frames, and outputting a video sequence descriptor at the last frame.

[0124] In the last year, some network architectures based on spatio-temporal fusion have appeared in the academic community, such as EchoVPR using ESN (Echo State Networks) based on recurrent neural network architecture to store intermediate states, and converting similarity calculation into probability calculation. However, the disadvantage is that each new application scenario needs additional training of the last layer network model, which limits the application scenarios of EchoVPR.

[0125] In order to improve the generalization ability of the model and avoid the need for separate training for each application scenario, SeqNet uses a time convolution network and a time aggregation method to generate sequence features, and constructs a video sequence geo-location retrieval architecture from coarse to fine.

[0126] Figure 2SeqNet is a state-of-the-art video geo-localization architecture based on temporal convolution.

[0127] As shown in Figure 2 For the current input video sequence, SeqNet first extracts global descriptors for each image, then splits into two branches, one branch uses temporal convolution to generate global descriptor for the video sequence and performs fast filtering in the video sequence dataset to get the top 3 most similar sequences; the other branch uses fully connected layer and L2 normalization to post-process the global descriptors of the images, then uses SeqMatch to match each image in the fast filtered result sequences, and finally calculates the optimal match according to the generated matching matrix.

[0128] However, SeqNet only uses image global descriptors to generate video sequence global descriptors, naturally losing a lot of information, ignoring the spatio-temporal correlation of features in the video sequence, in addition, the two branches of SeqNet are based on video global descriptors and image global descriptors respectively, both of which focus on overall global information, ignoring local features, so its retrieval accuracy is low.

[0129] The technical solution of the present disclosure aims to build an end-to-end relocalization system, input the established scene map and the current observation video, find the corresponding geographical location of the video in the scene map, and output the 6-DOF pose of the last frame of the video observation image.

[0130] When mapping, a video scene database is established to improve mapping accuracy, and effective loop detection is performed to correctly identify scenes that have been visited. When positioning, fast geo-location identification, scene merging, and positioning are performed to improve positioning accuracy.

[0131] The present disclosure improves the simultaneous localization and mapping (SLAM) system, mainly for video sequence geo-location identification and positioning, including the following points: 1. Quickly and accurately find candidate video sequences in the same geographical location as the current observation image in the massive video database. 2. According to the current video sequence, locate the geographical location in the scene map, i.e. output the 6-DOF pose. 3. Improve the geo-location identification accuracy and positioning accuracy when the scene appearance changes. 4. Improve the geo-location identification accuracy and positioning accuracy when there is foreground occlusion in the video image (cars, pedestrians, other dynamic objects). 5. For the problem of insufficient observation information for a single image, only local area of the scene, and occluded image, extend the simultaneous localization and mapping system to video sequences, perform spatio-temporal feature fusion and scene merging, and thus improve the accuracy and robustness of geo-location identification and positioning.

[0132] The technical problems solved by the technical solutions of the present disclosure include: 1. Scene appearance change: based on saliency weighting, not only the detailed texture features are focused on, but also the high-level semantic features are focused on; the pose optimization method of the present disclosure uses a deep learning network for image alignment, which is conducive to more robust feature extraction; the design of these two parts is mainly to solve the problems of high relocalization failure rate and inaccurate pose calculation when the scene appearance changes (lighting changes, viewing angle changes, seasonal changes, weather changes, local occlusions, etc.). 2. Local region correlation: in data preprocessing, traditional algorithms cannot solve the data labeling problem of images only locally related, and when performing geographic location recognition, the accuracy is not high only when part of the observation regions overlap. Therefore, the present disclosure extends it to video sequences, proposes a video scene spatio-temporal region matching algorithm and a scene merging algorithm, and designs a data preprocessing method based on common observation proportion.

[0133] Some embodiments of the present disclosure construct a new network system: a video geographic location recognition and positioning system (HAPOR) architecture from coarse to fine is constructed, which comprehensively considers image global information, salient region information and local point information, and the retrieval accuracy increases in turn, not only focusing on detailed texture features, but also focusing on high-level semantic features; first, candidate images are quickly retrieved from massive images, and then refined screening and optimization sorting are performed.

[0134] Some embodiments of the present disclosure fuse the SLAM system: the above HAPOR system is fused with the traditional ORB-SLAM2 system, which is used in the process of synchronous mapping and positioning, the iterative pose updating algorithm is used to improve the mapping accuracy when mapping, and the spatio-temporal region matching algorithm makes the loop detection more accurate; the positioning accuracy is improved when positioning, and the robustness to scene appearance is improved.

[0135] Some embodiments of the present disclosure extend image->video: a spatio-temporal region matching algorithm based on saliency weighting is designed, which improves the recall rate of geographic location recognition and retrieval; the features of each frame in the video sequence are spatio-temporally fused, and the robust region features under multi-view changes are selected. At the same time, the saliency image is obtained by the deep learning network, the high-confidence regions are selected for geographic location recognition, and the robustness to scene appearance changes is improved.

[0136] Some embodiments of the present disclosure improve the video sequence scene merging: the video scenes with common observations are merged, thereby removing the repeated redundant information, retaining the independent information of each, and providing more prior information for subsequent geographic location positioning.

[0137] The present disclosure uses a deep learning network for geographic location recognition and retrieval, so data preprocessing is needed for the collected video sequences, mainly video sequence segmentation and data labeling.

[0138] Segmentation of video sequence: For the input video stream, first extract part of the frames as key frames, so that the observed overlap area between key frames is within a reasonable range, to prevent the image overlap degree being too high and causing waste of computing resources. The extraction criteria of key frames can be based on the observed area overlap percentage, the number of inlier points of feature point matching, the geographical position relationship of two frames (such as translation distance, rotation angle, GPS coordinate distance), etc. Select M frames from the extracted key frames to form a video sequence. Different video sequences can be equal length or unequal length, and the parameter is between [3, 15]. The construction strategy of the video sequence can be continuous M key frames, or key frames are selected at equal intervals, or the common view relationship between key frames is determined by feature point matching and optical flow method, so as to dynamically select key frames with long time interval to construct video sequences.

[0139] Video sequence geographical position data labeling: It is mainly to label whether two video sequences are in the same geographical position, but the traditional discriminant labeling method according to GPS distance is too rough, and cannot effectively label scenes that are only partially related, thereby introducing too much dirty data. Therefore, the present disclosure designs a video sequence geographical position similarity labeling method based on video scene common observation ratio, which converts the original 0-1 discriminant problem of "whether in the same geographical position" into a probabilistic problem of "the possibility of two video sequences being in the same scene", and uses floating point numbers to measure more accurately.

[0140] Firstly, the present disclosure models the observation area of the camera.

[0141] Figure 3 is the camera observable area modeling (viewing cone model) used by the present disclosure.

[0142] As shown in Figure 3 , wherein HFOV is the horizontal field of view, VFOV is the vertical field of view, and DFOV is the diagonal field of view, all of which are angle values, and the calculation of the camera intrinsic parameters is as follows:

[0143]

[0144] , wherein f is the focal length of the camera, p is the size of each pixel, and w and h are the width and height of the camera image, respectively. The camera shooting position is represented by the vertex coordinates (x, y, z) of the viewing cone, denoted as E, and the specific calculation method is as follows: first, convert the GPS latitude and longitude coordinates corresponding to the image to Beijing 54 plane rectangular coordinates, and take them as the x and y coordinate values of the camera vertex, and take the fixed height of the camera as the z coordinate value, with the unit of meters. The direction of the central ray of the viewing cone vertex E depends on the image observation angle obtained before. The width-height ratio of the bottom surface of the viewing cone is consistent with the width-height ratio of the shooting image.

[0145] The depth of the perspective cone should depend on the actual depth of the scene being filmed, for example, long shots and close-ups obviously have different scene depths. To determine the depth information of the filmed scene, the present disclosure first extracts key points from the image, the purpose being to sample the depth information of the key areas of the image, while removing the influence of areas with excessively large depth values such as the sky, tunnels, caves, etc. In order to prevent the extracted image feature points from being concentrated in a certain part of the image, so that the key points can better represent the depth information of each region of the image in the subsequent process, the extracted key points need to be homogenized, and the present disclosure sequentially performs grid extraction of key points, non-maximum suppression, and k-ary tree region splitting screening, thereby extracting homogenized feature points. Then, the FastDepth (Fast Monocular Depth Estimation) algorithm is used to estimate the depth of the observed image, and the homogenized key points are mapped to the depth map. In order to further eliminate outliers of the key point depth, the present disclosure uses the percentile method to filter the depth values, sorts the depth values at the key points in ascending order and removes the key points at the front and back 20%, calculates the average value and maximum value of the depth values at the remaining key points, and the proportion of feature points with a depth value less than 5 meters. Finally, the calculated maximum value is taken as the height of the perspective cone (the vertical distance from the top point E to the bottom surface).

[0146] After obtaining the perspective cone model of a single frame image, the present disclosure uses OpenGL to repeat the above process for M frames of a video sequence, thereby constructing a 3D space body built by the perspective cones for each video sequence. Finally, the SDF space distribution function (Sign Distance Function) and the Monte Carlo algorithm are used to calculate the intersection-over-union of the 3D space bodies corresponding to the two video sequences. That is, in OpenGL, RN3D points are randomly sampled, and if the SDF space distribution function returns less than or equal to 0, it means that the 3D point is inside the object, otherwise it means that the 3D point is outside the object. Therefore, as long as the result of any SDF function is less than or equal to 0, it means that the space point is located in the perspective cone intersection region; if the results of both SDF functions are less than or equal to 0, it means that the space point is located in the perspective cone intersection region. Finally, the ratio of the number of sampling space points located in the perspective cone intersection region to the number of sampling space points located in the perspective cone intersection region is calculated, and the obtained intersection-over-union is taken as the common observation proportion of the two video sequences, i.e. the corresponding scene similarity.

[0147] It should be noted that this process is performed offline as part of data preprocessing, and the final visual geographic positioning system does not need to perform this step in real time.

[0148] Figure 4 is a flowchart of the method of video scene mapping and positioning of an embodiment of the present disclosure.

[0149] Reference Figure 4 In some embodiments of the present disclosure, the method S1000 of video scene mapping and positioning of the present disclosure comprises:

[0150] S100, constructing an image pyramid for a current observation image of a current observation video sequence, obtaining an initial pose, a confidence pyramid and a dense feature map of the current observation image based on the image pyramid, performing iterative pose optimization in a scene map based on the initial pose, the confidence pyramid and the dense feature map of the current observation image and obtaining an optimized pose to obtain a scene map comprising the optimized pose of the current observation image;

[0151] S200, performing fusion processing on the depth features of all frame images of the current observation video sequence to obtain a global descriptor of video spatio-temporal feature fusion of the current observation video sequence;

[0152] S300, performing video geographical location recognition using the global descriptor to obtain a plurality of initial candidate video sequences (K1) with the highest geographical location similarity to the current observation video sequence from a pre-constructed video sequence database, performing local region feature extraction on all frame images of the current observation video sequence to obtain region descriptors of a plurality of regions of each frame image;

[0153] S400, performing region matching based on the region descriptors on each frame image of the current observation video sequence and each frame image of each initial candidate video sequence, obtaining a spatial consistency degree of each initial candidate video sequence and the current observation video sequence based on the matching result, and further obtaining a plurality of final candidate video sequences (K2) corresponding to the spatial consistency degree from the plurality of initial candidate video sequences;

[0154] S500, for each final candidate video sequence, obtaining a co-view image cluster of each frame image in the scene map to fuse each independent final candidate video sequence into a single scene, obtaining a pose of the current observation image, i.e. a camera pose, based on the feature matching results of all 3D map points of each co-view image cluster in the scene map and the feature points of the current observation image;

[0155] S600, performing iterative pose optimization from low resolution images to high resolution images based on the image pyramid based on the pose of the current observation image to perform scene matching of the current observation image and each single scene to obtain a geographical location positioning result of the current observation image.

[0156] In some embodiments of the present disclosure, in step S100, further comprising: performing feature extraction on the image pyramid based on a convolutional neural network with an "encoder-decoder" structure to obtain a feature pyramid.

[0157] In some embodiments of the present disclosure, in step S100, the initial pose of the current observation image is obtained based on the image pyramid, including: S1102, performing image geographical location retrieval in the pre-constructed video sequence database based on the image pyramid to obtain a plurality of candidate images with the highest similarity to the geographical location of the current observation image; and S1104, obtaining the initial pose of the current observation image based on the co-visibility relationship of the plurality of candidate images.

[0158] In some embodiments of the present disclosure, in step S100, the confidence pyramid of the current observation image is obtained based on the image pyramid, including: performing confidence calculation on the feature pyramid to obtain the confidence pyramid.

[0159] In the confidence pyramid, different layers represent the confidence of regions at different scales, and each layer corresponds to the saliency of the corresponding position of the image pyramid.

[0160] In some embodiments of the present disclosure, the confidence calculation on the feature pyramid includes: S1202, inputting the feature pyramid into an SK-Net network based on a self-attention mechanism; and S1204, inputting the output of the SK-Net network as input and performing convolution processing on the first convolution layer to obtain the confidence pyramid.

[0161] In some embodiments of the present disclosure, in step S100, the dense feature map of the current observation image is obtained based on the image pyramid, including: performing dense feature calculation on the feature pyramid to obtain the dense feature map of the current observation image.

[0162] In some embodiments of the present disclosure, the dense feature calculation on the feature pyramid includes: inputting the feature pyramid into the second convolution layer to perform convolution processing to obtain the dense feature map of the current observation image.

[0163] In some embodiments of the present disclosure, preferably, further comprising: outputting the current observation image and its corresponding confidence image to obtain a visual result, and outputting a scene map including the optimized pose of the current observation image to obtain a visual result.

[0164] In some embodiments of the present disclosure, preferably, the pre-constructed video sequence database includes a plurality of video sequences, and the video sequences are obtained based on the following method:

[0165] The acquired video stream is subjected to key frame image extraction to obtain video sequences composed of key frame images, and then a plurality of video sequences are obtained.

[0166] In some embodiments of the present disclosure, the video sequence database includes the co-visibility relationship of each key frame image, and the co-visibility relationship is obtained based on the following method:

[0167] extracting scene depth information of all key frame images of each video sequence to construct a 3D view space body based on the scene depth information of the key frame images of each video sequence;

[0168] judging the coincidence degree of the 3D view space body of each video sequence to obtain the scene similarity between the corresponding scenes of each video sequence;

[0169] obtaining the co-view relationship of each key frame image based on the scene similarity between the corresponding scenes of each video sequence.

[0170] For step S100 of the present disclosure, in some embodiments of the present disclosure, it is realized by a hierarchical feature alignment and projection optimization geographic location positioning (HAPOR) network module designed by the present disclosure.

[0171] Figure 5 is a hierarchical feature alignment and projection optimization geographic location positioning (HAPOR) network module diagram of the present disclosure.

[0172] As shown in Figure 5 . The HAPOR network is divided into three modules in whole—backbone network module, hierarchical multi-task network module and pose iterative updating module. The input is the current observation image and the scene map, and the output is the 6-DOF (Degree Of Freedom) pose of the current observation image (i.e. camera).

[0173] For the current observation image, first generate a three-layer feature pyramid by the general backbone network module. Then enter the hierarchical multi-task network module, which is divided into three branches to obtain the initial pose, confidence pyramid and dense feature map respectively. The first branch mainly performs geographic location recognition and retrieval according to the image background to obtain candidate images in the same geographic location as the current observation image, and then calculates the initial pose according to the co-view relationship of the candidate images. The second branch mainly obtains the confidence pyramid corresponding to the feature pyramid according to the self-attention mechanism. The third branch is used to generate a dense feature map, each pixel of which corresponds to a feature vector. The output of the hierarchical multi-task network module will be passed as input to the iterative pose updating module. The iterative pose updating module will perform coarse-to-fine pose calculation and update, and finally output the 6-DOF pose of the current camera as the result of visual geographic location positioning.

[0174] (1) Backbone network module

[0175] The backbone network module of HAPOR is a general shared computing module. For a current observation image, firstly, image pyramid transformation is performed to obtain a three-layer image pyramid. The size of the bottom layer of the pyramid is the size of the input image, and the size of the upper layer of the pyramid can be one quarter of the next layer of the pyramid. The image pyramid transformation can make the model more robust to the perspective transformation in the scene, thereby improving the accuracy of the geographic position positioning in the scene of rotation, rapid motion and the like.

[0176] The obtained image pyramid is input into a convolutional neural network of an “encoder-decoder” structure to obtain a three-layer feature pyramid. The width and height of the size of the feature pyramid are consistent with those of the image pyramid, but the number of channels can be expanded to 32, 128 and 128 respectively. The Unet network based on the VGG19 feature extractor and pre-trained on the ImageNet is selected as the “encoder-decoder” module in the backbone network module.

[0177] (2) Hierarchical multi-task network module

[0178] The input of the hierarchical multi-task network module is the feature pyramid obtained by the backbone network module, which is then divided into three branches. The first branch is an initial pose calculation branch. The branch is input from the encoder of the backbone core network module, and after passing through another separate Unet decoder structure, the “geographic position-based image geographic position retrieval module” is used to retrieve five candidate images with the highest similarity to the current observation image from the key frame database during mapping, and then the “initial pose calculation module based on the co-view relationship” is used to calculate the initial pose R0 and t0.

[0179] The second branch is a confidence calculation branch. The branch is composed of a SK-Net network based on a self-attention mechanism and a 1x1 convolution layer. The output is a confidence pyramid corresponding to the feature pyramid. The width and height of the size of the confidence pyramid correspond to the size of each layer of the image pyramid, but the number of channels is 1. Each pixel represents the saliency of the corresponding position of the image pyramid. The saliency of the image corresponding to the bottom layer of the pyramid is shown in Figure 6 Figure 6 is the current observation image and its corresponding dense feature map and confidence map of an embodiment of the present disclosure.

[0180] As shown in Figure 6 , the right side is a confidence label. The reason why the SK-Net based on the self-attention mechanism is selected to generate the confidence map is to better cope with the perspective change and rapid motion. The third branch obtains a dense feature map corresponding to the feature pyramid through a 1x1 convolution, as shown in Figure 6

[0181] (3) Video scene mapping and positioning system fusion process

[0182] ​​ORB-SLAM2 is the most widely used SLAM system in the industry for autonomous driving applications. It contains three core threads. The main thread uses feature point method to minimize re-projection error to achieve tracking and positioning. The local optimization thread is responsible for the management and local BA (Bundle Adjustment) optimization of the local map. The loop detection thread is responsible for detecting image geographical location retrieval and correcting the cumulative drift error through global BA optimization.

[0183] The present disclosure combines the HAPOR model in Figure 5 with ORB-SLAM2 and applies it to video sequences to construct a new video scene mapping and positioning system. First, the DBoW (Distributed Bag of Words) model responsible for image geographical location retrieval in the loop detection thread is replaced by the video geographical location identification and retrieval model designed by the present disclosure (described below). Meanwhile, in the main thread under the localization mode (State = Localization), the original repositioning method based on feature points is replaced by the HAPOR model designed by the present disclosure (as shown in Figure 5 ). Any method can be selected for tracking and mapping during mapping, but each map point must be stored with the feature vector on the dense feature map in the scene map. In order to prevent cumulative drift error, the video geographical location identification and retrieval model designed by the present disclosure is used for loop detection. When it is found that the current scene has been visited before, global BA (Bundle Adjustment) optimization is immediately started to correct the map point coordinates and trajectory pose, thereby eliminating the cumulative error.

[0184] The video scene mapping and positioning system designed by the present disclosure can use any feature point method for mapping during the mapping phase, but it needs to additionally use the HAPOR network shown in Figure 5 for iterative pose optimization. The input of the iterative pose module is the initial pose R0, t0, confidence pyramid, hierarchical dense features, and scene map. For the set of co-visible clusters obtained in the scene merging module, the following algorithm process is performed in order from high to low according to the cluster score size. First, all 3D map points corresponding to the images in the co-visible cluster are taken out from the scene map to form a map point set PM. Since the map points may be observed by multiple key frame images in the co-visible cluster, it is necessary to perform a de-duplication process. Next, according to the initial pose R0, t0obtained by the scene merging module, the 3D map points in the map point set PM are projected onto the image coordinate system of the current observation image. By judging whether the projected points are located within the current observation image, the 3D map points located within the current observation image are selected to form a visible map point set VM.

[0185] (4) Coarse-to-fine geographical location positioning

[0186] This part is part of "scene map establishment based on iterative pose optimization", and also "video geolocation positioning from coarse to fine", the present disclosure relates to video scene mapping and positioning, mapping first and then positioning.

[0187] The projection point of each map point in the visible map point set VM corresponding to the current observation image is converted into a dense feature map coordinate, so as to obtain the corresponding dense feature. It should be noted that, since the projection point is not necessarily an integer in the image coordinate system, when obtaining the corresponding dense feature, the feature map needs to be linearly interpolated twice. At the same time, since the present disclosure performs image feature alignment of the current observation image and the candidate scene, each map point may be observed by multiple images in the candidate scene, and thus corresponds to multiple projection features. For each map point, the present disclosure defines the projection feature alignment residual as shown in the following formula.

[0188]

[0189] Where r i is the projection feature alignment residual of the i-th map point in the visible map point set VM; H is the dense feature of the current observation image corresponding to the map point; H' k is the projection feature corresponding to the k-th key frame image in the common view cluster to which the map point belongs; w i,k is the product of the confidence weight on the confidence map corresponding to the projection feature and the dense feature, respectively; ||·||2 represents the Euclidean distance.

[0190] The above formula describes the image feature alignment process based on saliency weighting. The confidence map generated by the HAPOR hierarchical multi-network module of the present disclosure weights the features, so that the model filters out foreground dynamic objects and pays more attention to features with stronger image saliency. Finally, the overall objective function can be defined according to the following formula.

[0191]

[0192] Where p is a robust kernel function, E(R,t) is the objective function of the current observation image pose (R,t) calculated, which converges the original multi-stage error based on feature point method, so that the model can use gradient descent algorithm for global error unified optimization.

[0193] Then, the LM algorithm is used for pose updating from coarse to fine. Unlike Pixloc and other algorithms, in order to improve the robustness to view angle changes and fast motion, the present disclosure performs image pyramid transformation at the beginning, so in the pose updating module, the present disclosure designs an iterative pose updating algorithm from coarse to fine. As Figure 5As shown, the disclosure adopts a coarse-to-fine pose iteration process of an inverted pyramid. The projection feature alignment residual is calculated according to formula (1) for the bottom layer of the pyramid at the beginning. The calculated pose will be used as the initial value for the pose iteration of the next layer of the pyramid. Finally, the pose of the top layer of the pyramid is updated. Thus, the K-round pose iteration process actually performs 3xK times of pose updating. The main design idea of the algorithm is to first focus on the alignment of the overall features, obtain a relatively accurate pose, and then perform projection alignment on the relatively fine features to correct the camera pose in a small way. That is, the low-resolution features are mainly responsible for the robustness of pose prediction because they contain more semantic information and have a larger receptive field, while the high-resolution features are responsible for refining the pose accuracy.

[0194] In each iteration process, the pose updating amount is calculated using the LM (Levenberg-Marquardt) algorithm, that is, the objective function in formula (2) is minimized. The derivative thereof is as follows:

[0195]

[0196] where δ represents the current pose, E(R, t) is the objective function described in formula (2), i is the map point label, k is the candidate image label, r is the projection feature alignment residual, p i q is the projection coordinate of the i-th map point projected into the current observation image coordinate system, F q is the dense feature map corresponding to the current observation image. Then, all the projection feature alignment residuals are stacked, denoted as matrix R, and all the confidence weights w i,k are also stacked, denoted as W, as shown in formula (4).

[0197] W = diag i,k (w i,k × ρ') (4)

[0198] where diag represents a diagonal matrix, and ρ' represents the derivative of the robust kernel function. According to the LM algorithm, the Lagrange function is constructed, and the incremental equation is solved to obtain the pose correction amount Δδ as follows:

[0199] Δδ = - (J T WJ + λ diag (J T WJ)) -1 J T WR (5)

[0200] where the parameter λ is a hyperparameter, J is the Jacobian matrix, and T is the transpose symbol. After obtaining the pose updating amount, the camera pose is updated according to formula (6).

[0201]

[0202] wherein (R + ,t + ) is the updated pose, ^ is the anti-symmetrization operation. Then the set of map points is projected again according to the updated pose, and a new set of visible map points is obtained, and the projection feature alignment residual is recalculated, and then the above operation is repeated, and the pose is updated iteratively until the pose update amount Δδ is small enough, and it is considered that the algorithm converges. The pose (R, t) at the final convergence is taken as the 6DOF pose of the current observation image, that is, the result of the geographic position positioning based on the iterative pose updating algorithm. The result is used as the output pose during mapping, thereby improving the accuracy of the trajectory, enhancing the robustness to changes in the appearance of the scene, and reducing the interference of foreground occlusion on the algorithm.

[0203] (5) Video scene mapping and positioning system running parameter setting and visualized result

[0204] In order to make the SLAM system more real-time, in some embodiments of the present disclosure, the observation image is scaled to a resolution size of 640x480. It should be noted that when the subsequent map points are projected from the world coordinate system to the image coordinate system, the size needs to be converted according to the camera intrinsic parameter. The present disclosure uses a pinhole imaging model with radial distortion as the camera model for testing, and ignores the influence of small tangential distortion. In addition, in order to ensure the real-time performance of the system, the maximum number of iterations K=30 is set in the iterative pose calculation module of HAPOR, and the running of the fused SLAM system is as shown in Figure 7 . Figure 7 is the running visualized result of the video scene mapping and positioning system designed by the present disclosure.

[0205] Referring to Figure 7 , the left graph is the scene map and the positioning position, and the right graph is the current observation image and the confidence image in the first row, the ORB-based repositioning matching in the second row, and the HAPOR initial pose matching result in the third row. The images on the left of the second row and the third row are the Top 1 candidate images obtained by the geographic position recognition and retrieval. As can be seen from Figure 7 , the current test scene is in spring, and the scene map used is established in autumn, and it can be seen that the appearance of the scene has changed greatly. In this case, the number of ORB feature point matches is small and there are obvious incorrect matching pairs, while the video scene mapping and positioning system designed by the present disclosure has a large number of matching pairs with strong consistency.

[0206] Figure 8 is a flowchart of the iterative pose optimization and optimized pose obtained in the scene map based on the initial pose of the current observation image, the confidence pyramid and the dense feature map of an embodiment of the present disclosure.

[0207] Based on the above description, in step S100 of the present disclosure, the initial pose of the current observation image, the confidence pyramid and the dense feature map are used to perform iterative pose optimization in the scene map and obtain an optimized pose, preferably including:

[0208] S1502, obtaining the 3D map points of each candidate image in the scene map of the multiple candidate images with a common view relationship, i.e., the common view image cluster, to form a 3D map point set;

[0209] S1504, projecting the 3D map points in the 3D map point set to the image coordinate system of the current observation image to determine whether each 3D map point is located within the current observation image;

[0210] S1506, obtaining a visible map point set based on the 3D map points located within the current observation image;

[0211] S1508, converting the projection points of each map point in the visible map point set in the current observation image into dense feature coordinates in the dense feature map to obtain the dense features corresponding to each map point in the visible map point set;

[0212] S1510, performing weighted processing on the dense features corresponding to each map point in the visible map point set and the multiple projection features of each map point on the current observation image based on the confidence pyramid corresponding to the current observation image to highlight the salient features of the current observation image, so as to align the overall image features of the current observation image and the candidate images;

[0213] S1512, taking the pose of the current observation image after the overall image feature alignment as an initial value, performing iterative pose optimization from a low-resolution image to a high-resolution image based on the image pyramid to obtain an optimized pose.

[0214] Preferably, in step S1512, taking the pose of the current observation image after the overall image feature alignment as an initial value, performing iterative pose optimization from the image features of a low-resolution image to the image features of a high-resolution image based on the image pyramid, including:

[0215] In turn, based on the image features of a low-resolution image to the image features of a high-resolution image based on the image pyramid, the following iterative pose optimization process is performed:

[0216] Based on the confidence of the confidence pyramid corresponding to the current resolution image, the dense features corresponding to each map point in the visible map point set and the multiple projection features of each map point on the current resolution image are weighted processed to highlight the salient features of the current resolution image, so as to align the image features of the current resolution image and the candidate images.

[0217] In step S1512, the iterative pose optimization process is performed based on the LM algorithm.

[0218] Figure 9 is a flowchart of obtaining a global descriptor of video spatio-temporal feature fusion of a current observation video sequence according to an embodiment of the present disclosure.

[0219] Reference Figure 9 In some embodiments of the present disclosure, the step S200 described above is to fuse the deep features of all frame images of the current observation video sequence to obtain a global descriptor of video spatio-temporal feature fusion of the current observation video sequence, including the following steps:

[0220] S2102, performing frame-by-frame feature extraction on all frame images of the current observation video sequence to obtain corresponding feature maps;

[0221] S2104, stacking the feature maps of all frame images in the channel dimension, and enhancing the features of each frame image to fuse the features of each frame image in the time domain, to obtain the feature maps of each frame image fused in the time domain;

[0222] S2106, performing a stitching operation on the feature maps of each frame image fused in the time domain to obtain a video sequence feature map (F);

[0223] S2108, performing residual merging and spatio-temporal feature aggregation on the video sequence feature map (F) to generate a global descriptor.

[0224] Preferably, the global descriptor is represented by a high-dimensional vector.

[0225] The step S200 of the present disclosure performs video spatio-temporal feature aggregation.

[0226] Unlike the method of generating a video global descriptor using an image global descriptor such as SeqNet, the present disclosure designs a brand-new network structure, TemproalVLAD, which does not need the complex step of generating an image global descriptor for each image in a video, but directly fuses the deep features of each image in a video sequence together. This not only reduces the time-consuming of calculation, but also helps to retain the unique observation information of each frame image and remove the redundant observation information between frames.

[0227] Figure 10 is a network structure diagram of TemproalVLAD designed by the present disclosure. As Figure 10As shown, its input is an M-frame video sequence, and its output is a global description of the video sequence. The basic design idea of ​​the TemproalVLAD network is to cluster features within the same video sequence together, thereby better fusing spatiotemporal features, preserving the unique observation information of each frame, and removing redundant observation information between frames, thus finding a more general and stable global representation of the video sequence. Furthermore, representing a video sequence with a high-dimensional vector not only facilitates fast retrieval but also allows for more efficient storage.

[0228] In some embodiments of this disclosure, for each image in a video sequence, feature extraction is first performed. The feature extraction network includes, but is not limited to, commonly used deep learning backbone networks such as VGG, UNet, ResNet, RegNet, AlexNet, GoogLeNet, and MobileNet. To prevent redundant computation, this disclosure uses HAPOR (… Figure 5 By using the VGG feature encoding network shown in the figure to extract features, each image will obtain a corresponding feature map.

[0229] After feature encoding for each image in the video sequence, the resulting feature maps are stacked along the channel dimension and then input into the CSE channel self-attention enhancement module. It's important to note that this is not a true self-attention mechanism, but rather an enhancement of features from different images within the same video sequence. The goal is to temporally fuse features extracted from single frames, thereby updating repeatedly observed features in the video sequence and making the model more robust to changes in viewpoint. Finally, the feature map output from the CSE attention module serves as input to the TemporalVLAD layer, where residual merging and spatiotemporal feature aggregation generate the global descriptor for the entire video sequence.

[0230] like Figure 10 As shown, the TemproalVLAD layer first concatenates the feature maps of the same video sequence along the long side to obtain the concatenated feature map F. Then, it performs pointwise convolution on the feature map F using a 1×1 convolution kernel and exponentially normalizes the result using Softmax to obtain the result a.

[0231] To ensure rapid network convergence, pre-computation is required on a large video dataset. Each video sequence generates a concatenated feature map F, which essentially consists of dense feature points, using the method described earlier. Then, all feature points from all feature maps are extracted, and unsupervised clustering is performed using the K-means++ clustering algorithm to obtain K cluster centers. It's important to note that this step is performed offline and can be considered as the initialization of network parameters.

[0232] With the cluster centers, in the residual calculation module of the Temporal VLAD layer, the distance between each point in the feature map F and the K cluster centers is calculated respectively, and the result a output by the exponential normalization unit is used as the weight for weighted summation, thereby obtaining K vectors corresponding to the global representation of the K clusters. Then, the vectors of each cluster are L2 normalized, and then the vectors of the K clusters are spliced together and globally L2 normalized, and finally a high-dimensional vector is obtained as the global descriptor of the entire video sequence.

[0233] Optionally, the obtained video descriptor can be reduced in dimension using principal component analysis (PCA) or a 1x1 convolution kernel, thereby being more efficiently stored.

[0234] The steps S300 and S400 of the present disclosure perform spatial-temporal region matching based on saliency weighting.

[0235] Figure 11 is a flowchart of obtaining the region descriptors of the multiple regions of each frame image according to an embodiment of the present disclosure.

[0236] Reference Figure 11 In some embodiments of the present disclosure, in step S300, local region feature extraction is performed on all frame images of the current observation video sequence to obtain the region descriptors of the multiple regions of each frame image, including:

[0237] S3102, performing feature enhancement and feature clustering processing on each layer of the feature pyramid corresponding to the image pyramid of the frame image to obtain multiple (S) cluster centers;

[0238] S3104, performing weighted residual calculation on each feature of the feature pyramid and each cluster center to obtain a residual pyramid corresponding to the feature pyramid;

[0239] S3106, for each layer of the residual pyramid, using a sliding window of a preset size (p x p) to perform region extraction, obtaining multiple region blocks of each layer of the residual pyramid, and obtaining a residual integral image of each region block;

[0240] S3108, obtaining the region descriptors of the multiple regions of each layer of the feature pyramid corresponding to the residual pyramid based on the residual integral image.

[0241] Figure 12 is a flowchart of obtaining multiple final candidate video sequences corresponding to the degree of spatial consistency from multiple initial candidate video sequences according to an embodiment of the present disclosure.

[0242] Reference Figure 12In the method for video scene mapping and positioning of some embodiments of the present disclosure, in step S400, the region descriptors of each frame image of the current observation video sequence and the region descriptors of each frame image of each initial candidate video sequence are matched based on region descriptors, the spatial consistency degree of each initial candidate video sequence and the current observation video sequence is obtained based on the matching result, and then one or more than two final candidate video sequences corresponding to the spatial consistency degree are obtained from the plurality of initial candidate video sequences, which includes:

[0243] S4102, the distance between all region descriptors of each frame image (Tm) of the current video observation sequence and all region descriptors of other frame images (Tn) is calculated to obtain a distance matrix (D), and a region matching set (Si) between different frame images (Tm and Tn) is obtained based on the distance matrix (D);

[0244] S4104, the region descriptor of each frame image (Tm) in the region matching set (Si) with the minimum average distance to all region descriptors of other frame images (Tn) is taken as the representative region descriptor of each frame image to obtain the representative region descriptor of the current observation video sequence;

[0245] S4106, the region descriptors of each frame image of the current observation video sequence and the region descriptors of each frame image of each initial candidate video sequence are matched based on the region descriptors to obtain a preliminary matching result;

[0246] S4108, the preliminary matching result is weighted matched based on the confidence pyramids of the frame images of the current observation video sequence and each initial candidate video sequence to obtain the spatial consistency degree (spatial consistency score) between each two frame images between the current observation video sequence and each initial candidate video sequence as the weighted matching result;

[0247] S4110, the spatial consistency degree between the current observation video sequence and each initial candidate video sequence is obtained based on the spatial consistency degree between each two frame images between the current observation video sequence and each initial candidate video sequence, and then a plurality of final candidate video sequences corresponding to the spatial consistency degree are obtained from the plurality of initial candidate video sequences.

[0248] In some embodiments of the present disclosure, video geographical location recognition is first performed using a video global descriptor (mainly considering fast geographical location recognition, according to the information of the whole video), which is used for loop detection in the mapping process and candidate video retrieval in the positioning step, to obtain K1 candidate video sequences, and then the present disclosure designs Figure 13 the image region feature extraction network shown in the figure, so as to extract region features of each frame image of the video sequence (mainly considering accurate geographical location recognition, according to local region information).

[0249] Figure 13 is a schematic diagram of an image region feature extraction network structure designed by the present disclosure.

[0250] As shown in Figure 13 , the input of the region feature extraction network is the feature pyramid input by the backbone network module in the HAPOR( Figure 5 ) system. For each layer of the pyramid, first pass through a CSE (Spatial Squeeze and Channel Excitation) module, then by a 1x1 convolution layer for dimensionality, the purpose is to better subsequent division of features. The dimensionality of the feature after the upgrade enters the multi-scale clustering module for feature clustering, then each feature and S cluster centers calculate the weighted residual error, get the residual pyramid corresponding to the feature pyramid.

[0251] In the multi-scale clustering module, first use a 1x1 convolution kernel to perform point-by-point convolution on the input feature map, and use the Softmax function to perform exponential normalization on the result to get the weight a. Then take the value of each feature at each channel as its descriptor, calculate the descriptor distance between the feature and each cluster center, and use the weight a for weighting, so as to get the residual feature map. Different scale residual feature maps constitute a residual pyramid, whose dimension is as shown in Figure 13 .

[0252]

[0253] The calculation process of the multi-scale clustering module is shown in the above formula, where x i is the feature on the input feature map, s is the number of cluster, si is the cluster label to which the current feature belongs, c si is the cluster center of its 4, w and b are the convolution kernel parameters. When calculating the descriptor distance, the present disclosure preferably uses the L2 distance (Euclidean distance) function.

[0254] After obtaining the residual pyramid, first in each layer of the residual pyramid, use a sliding window of fixed size (p x p) to extract regions, get the region block of each layer of the pyramid. Then calculate the residual integral image of each region block. When using the sliding window algorithm to extract the region feature, the essence is to sum all the feature vectors in the current sliding window, and the resulting vector is taken as the descriptor of the region. Since there is a lot of repeated calculation in the sliding window summation process, the present disclosure uses integral image for optimization.

[0255] Figure 14 is an auxiliary schematic diagram of the integral image optimization algorithm of one embodiment of the present disclosure.

[0256] As shown in Figure 14As shown, when the sliding window summation algorithm is optimized using the integral image algorithm, only one pass of the feature map is needed, and the cumulative sum of the feature vector in the region formed by each point and the origin is accumulated in turn, denoted as Accumulate(Point). The cumulative sum of point 2 in the figure, Accumulate(2), is equivalent to the sum of the feature vectors in the two regions Ra and Rb.

[0257] In the integral image optimization algorithm, the feature vectors in each region do not need to be summed separately. For example, the cumulative sum of the feature vectors in region Rc is equivalent to Accumulate(3)-Accumulate(1), and the cumulative sum of the feature vectors in region 4 is equivalent to Accumulate(1)+Accumulate(4)-Accumulate(2)-Accumulate(3). In this way, the algorithm complexity of O(WxHxp) can be optimized to O(WxH), i.e., only one pass of the feature map is needed. Figure 13 2

[0258] The image region feature extraction network is finally reduced in dimension by a 1x1 convolutional layer and normalized by an L2 norm to obtain the region descriptors of the regions at each level of the pyramid.

[0259] It is worth noting that, unlike methods such as Patch-NetVLAD, the present disclosure uses image pyramid transformation, which is more robust to changes in viewing angle and rapid motion, and thus a fixed-size sliding window can be used in the image region extraction network to extract region features at different levels of the pyramid.

[0260] After extracting region features from each frame of the video sequence, the present disclosure designs a spatiotemporal region matching algorithm based on saliency weighting (as shown in Figure 15 Figure 15 is a flowchart of the spatiotemporal region matching algorithm based on saliency weighting of an embodiment of the present disclosure. Referring to Figure 15 , accurate geolocation recognition is achieved for the current observation video and K1 candidate videos obtained from the video global descriptor. The purpose of designing the spatiotemporal region matching algorithm based on saliency weighting is to enhance the robustness of the region features to changes in viewing angle, and to better apply dynamic occlusion in complex relocation scenarios.

[0261] Referring to Figure 15 ​​​For the current observed video sequence, firstly region extraction is performed. Then region feature tracking and matching of the self sequence is performed, the purpose is to use region descriptors which are robust under different view angles to update the original region descriptors. The region matching here can use the way of combining bidirectional matching and ratio test. For a video sequence with length M frames, firstly calculate the distance between all region descriptors of Tm frames and all region descriptors of Tn frames (m, n ∈ [1, M] and m ≠ n), thereby forming a distance matrix D, the element Dij in the matrix represents the distance between the ith region descriptor of Tm frames and the jth region descriptor of Tn frames.

[0262] The present disclosure still uses Euclidean distance to measure the distance of the descriptors. Then the region matching way based on bidirectional matching and ratio test is to select the element coordinates from the distance matrix D which satisfy the following formulas as the matching items:

[0263]

[0264] Wherein D ij k represents the element with the minimum distance value in the jth column of the distance matrix D, D i k j represents the element with the minimum distance value in the ith row of the distance matrix D, t is a threshold parameter, and the matching items (i, j) meeting the condition constitute the matching set P mn between Tm frames and Tn frames. ij The distance value D i is also stored in the matching set. It should be noted that region matching is performed on the extracted multi-scale hierarchical region features respectively, and only region matching is performed on region features of the same scale.

[0265] When all (m, n) combinations of the same video sequence are exhausted, the region i of each frame is matched with the regions j in several frames, recorded as a set S L = {i, j1, j2…, j i}. And the region j is matched with j' in several frames in its own frame, and the region descriptors meeting the following formula are selected to replace the original descriptors of all regions in the set S i , thereby making it more robust to view angle changes.

[0266]

[0267] Wherein x is a region belonging to S x , P x is the matching item set corresponding to the region x in the matching set of the frame where the region x is located, and D x is all D ijSet, D x For the above D ij The average value of the elements in the set. Choose S. i D x The descriptors corresponding to the region x' are set as S i A new region descriptor is generated for all regions in the video sequence. This descriptor will serve as the representative of this region in the video sequence for subsequent region matching between different video sequences. Since region x' has the smallest average distance to other matching regions in the video sequence, the features of region x' are considered more robust under different observation perspectives.

[0268] Next, temporal region matching is performed between different video sequences. Specifically, during video scene retrieval, for the currently observed video sequence Vqry and a video sequence Vref stored in the database when building the scene map, the descriptors representing the regions of each video sequence are used. Following the region matching method based on bidirectional matching and ratio testing, the region matching between each frame in Vqry and each frame in Vref is calculated sequentially. The region matching results are weighted using the Lth layer of the confidence pyramid. First, a p×p sliding window is used to extract region blocks from the confidence map at the Lth layer. Then, the average value of all pixels within each region block is calculated; this average value is used as the "regional saliency weight." Finally, for each pair of matched regions in the "regional matching pair set," the region matching weight is calculated using the following formula:

[0269]

[0270] Where sigA and sigB are the "regional saliency weights" of the two matching regions, and Rmw (Regional MatchWeight) is the regional matching weight. That is, on the one hand, the larger the product of the saliency of the regions, the larger the regional matching weight; on the other hand, the more similar the saliency of the regions, the larger the regional matching weight.

[0271] The spatial consistency score, weighted by saliency regions, is used to evaluate the region matching at each level of the pyramid. The aim is to measure the quality of region matching based on their spatial relationship, penalizing regions with few or disorganized matches. A higher spatial consistency score indicates higher similarity, resulting in the candidate image appearing earlier in the ranking. The spatial consistency score for the Lth level region matching is shown in the following formula.

[0272]

[0273] Among them, Rms L n represents the weighted score of the region matching at level L of the pyramid; L P represents the number of descriptors for the Lth layer of the pyramid; LFor the "set of region matching pairs" at level L of the pyramid; (r L ,c L ) is P L The row and column offsets of the matching center are stored in the middle; and Representing P respectively L In the set (r L ,c L The average values ​​of ) are the average row offset and average column offset of the matching center; i, j represent the set P. p The numbering during traversal; the dist(.) function is the distance function, and this disclosure uses absolute distance; max(.) is the maximum value function.

[0274] Calculate Rms for each level of the pyramid. L The weighted score for region matching between the current observed image and the candidate image is calculated using the following formula.

[0275]

[0276] Where Rms (Regional Match Score) is the weighted score for region matching across the entire image pyramid; w L Let w be the weight of the Lth level of the pyramid. i ∈[0,1].

[0277] After obtaining the spatial consistency score between every two frames of the two video sequences, the final spatial consistency score of the two video sequences is calculated using the following formula.

[0278]

[0279] Where Rvms (Regional Video Matching Score) represents the spatial consistency score between two video sequences; m and k are the spatial consistency scores of the currently observed video sequence V. qry and retrieving video sequence V ref Frames in Rms m,k The spatial consistency score for these two images is V. ref ∈{TopK1 video sequences obtained from the fast search branch}; λ m Let λ be the weight of the observed frame, and λ be the weight of the observed frame. m The selection strategy for the frame ∈(0,1] is as follows: the last frame in the observed video sequence is selected as the reference frame, and the weight of this frame is 1. The weights of other frames are exponentially decayed from the reference frame.

[0280] The video sequence spatial consistency score is mainly used to measure the consistency of the local area emission in the scene. In terms of semantics, for example, different floors of a building have different design styles. If these different style appearances are observed in another video sequence and the spatial arrangement order also has consistency, then the probability that the two scenes are in the same geographic location is large. It should be noted that the region matching algorithm designed in the present disclosure pays more attention to local regions such as texture details, and uses the confidence of the region for weighting, so as to pay more attention to the background regions with higher saliency and ignore dynamic objects in the foreground.

[0281] The spatiotemporal region matching algorithm based on saliency weighting can accurately identify the geographic location based on the spatial consistency score Rvms of the video sequence. The higher the spatial consistency score is, the higher the ranking is. Finally, the top K2 video sequences are selected as the result of geographic location identification.

[0282] The step S500 of the present disclosure performs multi-video geographic location identification and scene merging.

[0283] Figure 16 FIG. 1 is a flowchart of multi-video geographic location identification and scene merging according to an embodiment of the present disclosure.

[0284] Reference Figure 16 In some embodiments of the present disclosure, in step S500, for each final candidate video sequence, the co-view image cluster of each frame image in the scene map is obtained to fuse each independent final candidate video sequence into a single scene. The pose of the current observation image, i.e., the camera pose, is obtained based on the feature matching results of all 3D map points of each co-view image cluster in the scene map and the feature points of the current observation image, including:

[0285] S5102, for each final candidate video sequence, each frame image that has a co-view relationship in the scene map and a co-view ratio greater than or equal to a preset threshold (for example, > 30%) is taken out one by one to obtain a co-view image cluster that has a co-view relationship greater than or equal to the co-view ratio threshold with each frame image;

[0286] S5104, the images in each co-view image cluster are de-duplicated to fuse each independent final candidate video sequence into a single scene;

[0287] S5106, for each co-view image cluster after deduplication processing, screening images belonging to the initial candidate video sequence, and using the similarity between the current observation video sequence and the global descriptor of the initial candidate video sequence to which the screened image belongs as the score of the screened image, the score of the image not belonging to the initial candidate video sequence is 0, and the scores of the images in each co-view image cluster are accumulated to obtain the score of each co-view image cluster;

[0288] S5108, based on the scores of the respective co-view image clusters from high to low, the following steps are sequentially executed: taking all 3D map points of the co-view image cluster from the scene map; extracting feature points of the last frame of the current observation video sequence, i.e., the current observation image, to perform 2D-3D feature matching with the 3D map points; based on the feature matching result, solving the pose of the current observation image, i.e., the camera pose, and taking it as the initial pose of the co-view image cluster.

[0289] In step S500, for the current observation video sequence, the purposes of video scene mapping and positioning are to calculate the 6-DOF pose corresponding to the last frame image. Since single image or single video sequence has observation limitations, comprehensive scene information cannot be obtained, therefore, the disclosure first performs multi-video geographical location recognition, and performs scene merging according to the co-view relationship. Finally, video geographical location positioning of video-scene is performed (step S600).

[0290] The process of multi-video geographical location recognition is: for the current observation video sequence, first, the video global descriptor calculated by the TemporalVLAD network described above is used to perform fast geographical location recognition, to obtain K1 candidate videos with the highest similarity to the current observation video, mainly investigating the global information and overall scene feature similarity of the video; then, the K1 candidate video sequences are subjected to accurate geographical location recognition according to the space-time region matching algorithm described above, and finally, K2 video sequences with the highest spatial consistency score are selected as the result of multi-video geographical location recognition. Exemplarily, K1 is between [20, 50], and K2 is between [3, 20].

[0291] After multi-video geographical location recognition, the disclosure designs a video scene merging algorithm.

[0292] The process of scene merging: firstly, K2 video sequences obtained by multi-video geographical location recognition are taken out one by one, and each frame image that has a co-view relationship in the scene map and a co-view ratio > 30% is taken out. Then all the images are clustered according to the co-view relationship, that is, images with a visual relationship are divided into the same cluster. Then the images in each cluster are de-duplicated, and the co-view cluster at this time is equivalent to merging the original independent candidate video into a single scene. For each co-view cluster, the images belonging to the original K1 candidate video sequences are screened, and the similarity (i.e. 1-descriptor distance) between the current observation video sequence and the global descriptor of the video sequence to which it belongs is used as the score of the image. If the image does not belong to the original K1 candidate video sequence, the score is 0. The scores of the images in the co-view cluster are accumulated, and the value is taken as the score of the co-view cluster.

[0293] According to the order of the co-view cluster score from high to low, the following calculation is performed: all map points of the co-view cluster are taken out from the scene map. Then the feature points of the current observation image (i.e. the last frame of the current observation video sequence) are extracted, and 2D-3D feature matching is performed with the 3D map points. The matching results are filtered by the Ransac (Random sample consensus) algorithm to remove incorrect matches, and the camera pose is finally solved by the EPnP (Effective Perspective-n-Point) algorithm. The calculated pose will be used as the initial pose of the co-view cluster, and subsequent video geographical location positioning will be performed according to step S600. The difference is that when calculating the initial pose, the map point set corresponding to all images of the co-view cluster is used, while in the subsequent iterative pose optimization, the map points will be projected onto the image coordinate system of the current observation image according to the initial pose. The map points outside the three times the original size of the image are removed as a boundary, thereby reducing the number of map points that need to be iterated. The remaining map points will be used in S600 to update the iterative pose.

[0294] Step S600 of the present disclosure performs coarse-to-fine video geographical location positioning: based on the pose of the current observation image, the iterative pose optimization from low-resolution image to high-resolution image based on the image pyramid is performed to match the current observation image with each individual scene, and the geographical location positioning result of the current observation image is obtained.

[0295] The basic process of this step is basically the same as the process described above "(4) from coarse to fine geographic location positioning". This step is used in geographic location positioning. Unlike video scene mapping, in positioning, feature points can be extracted first, and then iterative pose optimization can be performed, just like in mapping. Alternatively (this disclosure adopts this approach, which is more real-time), the initial pose obtained from step S500 "multi-video geographic location recognition and scene merging" can be directly used for iterative pose optimization from coarse to fine.

[0296] However, it should be noted that the iterative pose optimization here locates the 6-DOF pose of the last frame of the current video sequence, and the previously described S200 "video spatio-temporal feature aggregation", S300 and S400 "spatio-temporal region matching based on saliency weighting", and S500 "multi-video geographic location recognition and scene merging" are all for the entire video sequence. Therefore, the overall logic is to first perform geographic location recognition, retrieval, and fusion of the entire spatio-temporal features, then perform scene merging on the candidate video sequences identified and retrieved to extract more features in the same geographic location video sequence, thereby providing more prior observation information. Then in the "coarse-to-fine video geographic location positioning" link, it is no longer "image-image" or "video-video" matching, but image of the last frame of the video - scene matching, thereby performing geographic location positioning.

[0297] In summary, the present disclosure unifies geographic location recognition and positioning into one network architecture, greatly reducing the computational complexity while strengthening the coupling relationship between the two. Overall, it has the advantages of low computational complexity, small repositioning latency, high positioning accuracy, robustness to external environment changes (such as changes in lighting, changes in viewing angle, changes in weather, etc.), and other advantages. Compared with the single-frame image-based mapping and positioning (SLAM) algorithm, the present disclosure has at least the following advantages: considering saliency regions, it is more robust to scene appearance changes; considering high-level semantic features, it has stronger interpretability and is less affected by occlusion and repetitive textures. By fusing more spatio-temporal features, it solves the problem of insufficient observation information from a single image and local relevance, while improving trajectory accuracy and scene correctness during mapping, and improving geographic location recognition accuracy and positioning pose accuracy during positioning.

[0298] The present disclosure also provides a video scene mapping and positioning device 1000, comprising:

[0299] The scene map establishing module 1002 performs: constructing an image pyramid for a current observation image of a current observation video sequence, obtaining an initial pose, a confidence pyramid and a dense feature map of the current observation image based on the image pyramid, performing iterative pose optimization in a scene map based on the initial pose, the confidence pyramid and the dense feature map of the current observation image and obtaining an optimized pose to obtain the scene map including the optimized pose of the current observation image;

[0300] The video spatio-temporal feature aggregation module 1004 performs: performing fusion processing on the depth features of all frame images of the current observation video sequence to obtain a global descriptor of the video spatio-temporal feature fusion of the current observation video sequence;

[0301] The spatio-temporal region matching module 1006 performs: performing video geographical location recognition using the global descriptor to obtain a plurality of initial candidate video sequences (K1) with the highest geographical location similarity to the current observation video sequence from a pre-constructed video sequence database, performing local region feature extraction on all frame images of the current observation video sequence to obtain region descriptors of a plurality of regions of each frame image; performing region matching based on the region descriptors on each frame image of the current observation video sequence and each frame image of each initial candidate video sequence, obtaining a spatial consistency degree of each initial candidate video sequence and the current observation video sequence based on the matching result, and further obtaining a plurality of final candidate video sequences (K2) corresponding to the spatial consistency degree from the plurality of initial candidate video sequences;

[0302] The multi-video geographical location recognition and scene merging module 1008 performs: for each final candidate video sequence, obtaining a co-view image cluster of each frame image in the scene map to fuse each independent final candidate video sequence into a single scene, and obtaining a pose of the current observation image, i.e., a camera pose, based on feature matching results of all 3D map points of each co-view image cluster in the scene map and feature points of the current observation image;

[0303] The video geographical location positioning module 1010 performs iterative pose optimization from a low-resolution image to a high-resolution image based on the image pyramid based on the pose of the current observation image to perform scene matching of the current observation image and each single scene to obtain a geographical location positioning result of the current observation image.

[0304] Figure 17 FIG. 1 is a structural schematic block diagram of an apparatus 1000 of a video scene mapping and positioning method according to an embodiment of the present disclosure.

[0305] The apparatus 1000 for video scene mapping and localization can include respective modules to perform the various or some of the steps in the above flowcharts. Thus, each or some of the steps in the above flowcharts can be performed by respective modules, and the apparatus can include one or more of these modules. A module can be one or more hardware modules specifically configured to carry out the respective steps, or implemented by a processor configured to perform the respective steps, or stored within a computer-readable medium for implementation by a processor, or by some combination.

[0306] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and overall design constraints. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0307] The bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0308] Any process or method described in flowcharts or otherwise described herein can be understood as representing a module, segment, or portion of code that includes one or more executable instructions for implementing specific logic functions or steps, and the scope of preferred embodiments of the present disclosure includes additional implementation in which the functions are performed in different orders, including substantially simultaneously, or in reverse order, according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments in the present disclosure can be implemented as a software program tangibly embodied in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via the memory and / or communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the above-described methods can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods by any other appropriate means (e.g., by means of firmware).

[0309] Logic and / or steps represented in flow diagrams or otherwise described herein can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

[0310] For the purposes of this specification, a "computer-readable storage medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable storage medium can specifically include the following, which are non-exhaustive examples: electrical connection (electrical device), portable computer diskette (magnetic device), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber device, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable storage medium can even be paper or other suitable medium upon which the program can be printed, as the program can be electronically captured, for example, via the optical scanner of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in the memory.

[0311] The present disclosure also provides an electronic device, comprising: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored by the memory, so that the processor or other hardware module performs the above-mentioned method for video scene mapping and positioning.

[0312] The present disclosure also provides a readable storage medium, in which execution instructions are stored, and the execution instructions are executed by a processor to implement the above-mentioned method for video scene mapping and positioning.

[0313] In the description of the present specification, the description of the term "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. It is not necessarily the same embodiment / way or example in the illustrative description of the above terms in the present specification. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples without contradiction.

[0314] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0315] The person skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made on the basis of the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A method for video scene mapping and localization, the method comprising: Comprise: S100, constructing an image pyramid for a current observation image of a current observation video sequence, obtaining an initial pose, a confidence pyramid and a dense feature map of the current observation image based on the image pyramid, performing iterative pose optimization in a scene map based on the initial pose, the confidence pyramid and the dense feature map of the current observation image and obtaining an optimized pose to obtain a scene map comprising the optimized pose of the current observation image; S200, fusing depth features of all frame images of the current observation video sequence to obtain a global descriptor of video spatio-temporal feature fusion of the current observation video sequence; S300, using the global descriptor for video geographical location recognition, obtaining a plurality of initial candidate video sequences with the highest geographical location similarity to the current observation video sequence from a pre-constructed video sequence database, performing local region feature extraction on all frame images of the current observation video sequence to obtain region descriptors of regions of each frame image; S400, performing region matching based on region descriptors on each frame image of the current observation video sequence and each frame image of each initial candidate video sequence, obtaining a spatial consistency degree of each initial candidate video sequence and the current observation video sequence based on the matching result, and further obtaining a plurality of final candidate video sequences corresponding to the spatial consistency degree from the plurality of initial candidate video sequences; S500, for each final candidate video sequence, obtaining a co-view image cluster in a scene map for each frame image to fuse each independent final candidate video sequence into an individual scene, obtaining a pose of the current observation image, i.e. a camera pose, based on feature matching results of all 3D map points of each co-view image cluster in the scene map and feature points of the current observation image; And S600, performing iterative pose optimization from low resolution images to high resolution images based on the image pyramid based on the pose of the current observation image to perform scene matching between the current observation image and each individual scene to obtain a geographical location positioning result of the current observation image.

2. The video scene mapping and positioning method according to claim 1, wherein Optionally, in step S100, further comprising: performing feature extraction on the image pyramid based on a convolutional neural network with an encoder-decoder structure to obtain a feature pyramid, Optionally, in step S100, obtaining the initial pose of the current observation image based on the image pyramid comprises: S1102, performing image geographical location retrieval in a pre-constructed video sequence database based on the image pyramid to obtain a plurality of candidate images with the highest geographical location similarity to the current observation image; and S1104, obtaining the initial pose of the current observation image based on co-view relationships of the plurality of candidate images, Optionally, in step S100, the confidence pyramid of the current observation image is obtained based on the image pyramid, including: performing confidence calculation on the feature pyramid to obtain the confidence pyramid; wherein different layers in the confidence pyramid represent region confidence of different scales, and each layer corresponds to the saliency of the image pyramid at the corresponding position, Optionally, the confidence calculation on the feature pyramid includes: S1202, inputting the feature pyramid into an SK-Net network based on a self-attention mechanism; and S1204, inputting the output of the SK-Net network as input and performing convolution processing on the first convolution layer to obtain the confidence pyramid, Optionally, in step S100, the dense feature map of the current observation image is obtained based on the image pyramid, including: performing dense feature calculation on the feature pyramid to obtain the dense feature map of the current observation image, Optionally, the dense feature calculation on the feature pyramid includes: inputting the feature pyramid into the second convolution layer to perform convolution processing to obtain the dense feature map of the current observation image.

3. The method of video scene mapping and localization of claim 2, wherein, In step S100, the optimized pose is obtained by performing iterative pose optimization in the scene map based on the initial pose of the current observation image, the confidence pyramid and the dense feature map, including: S1502, obtaining the 3D map points of each candidate image in the co-view image cluster having a co-view relationship in the scene map to form a 3D map point set; S1504, projecting the 3D map points in the 3D map point set onto the image coordinate system of the current observation image to determine whether each 3D map point is located within the current observation image; and S1506, obtaining a visible map point set based on the 3D map points located within the current observation image; S1508, converting the projection points of each map point in the current observation image in the visible map point set into dense feature coordinates in the dense feature map to obtain the dense features corresponding to each map point in the visible map point set; S1510, performing weighted processing on the dense features corresponding to each map point in the visible map point set and the multiple projection features of each map point on the current observation image based on the confidence pyramid corresponding to the current observation image to highlight the saliency features of the current observation image, to align the overall image features of the current observation image and the candidate image; and S1512, taking the pose of the current observation image after the overall image feature alignment as an initial value, performing iterative pose optimization of the low-resolution image to the high-resolution image based on the image pyramid to obtain the optimized pose.

4. The method of video scene mapping and localization of claim 3, wherein, In step S1512, taking the pose of the current observation image after the overall image feature alignment as an initial value, performing iterative pose optimization of the image features of the low-resolution image to the image features of the high-resolution image based on the image pyramid, including: The following iterative pose optimization process is performed based on image features of low resolution images of the image pyramid to image features of high resolution images in sequence: The confidence of the confidence pyramid corresponding to the current resolution image is used to weight the dense features corresponding to each map point in the set of visible map points and the multiple projection features of each map point on the current resolution image to highlight the salient features of the current resolution image, so as to align the image features of the current resolution image and the candidate image.

5. The video scene mapping and positioning method according to any one of claims 1 to 4, wherein, Optionally, in step S1512, the iterative pose optimization process is performed based on an LM algorithm, Optionally, the method further comprises: outputting the current observation image and the confidence image corresponding thereto to obtain a visualization result, and outputting a scene map including the optimized pose of the current observation image to obtain a visualization result, Optionally, the pre-constructed video sequence database includes a plurality of video sequences, and the video sequences are obtained based on the following method: performing key frame image extraction on the acquired video stream to obtain a video sequence composed of key frame images, and then obtaining a plurality of video sequences.

6. The method of video scene mapping and localization according to any one of claims 1-5, wherein, The video sequence database includes a mutual visibility relationship of each key frame image, and the mutual visibility relationship is obtained based on the following method: Performing scene depth information extraction on all key frame images of each video sequence to construct a 3D perspective space body based on key frame image scene depth information for each video sequence; Judging the degree of coincidence of the 3D perspective space body of each video sequence to obtain scene similarity between scenes corresponding to each video sequence; and Obtaining the mutual visibility relationship of each key frame image based on the scene similarity between scenes corresponding to each video sequence, Optionally, in S200, the depth features of all frame images of the current observation video sequence are fused to obtain a video space-time feature fused global descriptor of the current observation video sequence, including: in S2102, performing frame-by-frame feature extraction on all frame images of the current observation video sequence to obtain corresponding feature maps; in S2104, stacking the feature maps of all frame images in the channel dimension, enhancing the features of each frame image to fuse the features of each frame image in the time domain, and obtaining time domain fused feature maps of each frame image; in S2106, performing a stitching operation on the time domain fused feature maps of each frame image to obtain a video sequence feature map; and in S2108, performing residual merging and space-time feature aggregation on the video sequence feature map to generate the global descriptor. Optionally, the global descriptor is represented using a high-dimensional vector. Optionally, in step S300, local region feature extraction is performed on all frame images of the current observation video sequence to obtain region descriptors of multiple regions of each frame image, including: S3102, performing feature enhancement and feature clustering processing on each layer of a feature pyramid corresponding to an image pyramid of a frame image to obtain multiple cluster centers; S3104, performing weighted residual error calculation on each feature of the feature pyramid and each cluster center to obtain a residual error pyramid corresponding to the feature pyramid; S3106, using a sliding window of a preset size to perform region extraction on each layer of the residual error pyramid to obtain multiple region blocks of each layer of the residual error pyramid, and obtaining a residual error integral graph of each region block; and S3108, obtaining region descriptors of multiple regions of each layer of the feature pyramid corresponding to the residual error pyramid based on the residual error integral graph. Optionally, in step S400, region matching based on region descriptors is performed on each frame image of the current observation video sequence and each frame image of each initial candidate video sequence, and a spatial consistency degree between each initial candidate video sequence and the current observation video sequence is obtained based on a matching result, and one or more than two final candidate video sequences corresponding to the spatial consistency degree are obtained from the multiple initial candidate video sequences, including: S4102, calculating distances between all region descriptors of each frame image of the current observation video sequence and all region descriptors of other frame images to obtain a distance matrix, and obtaining a region matching set between different frame images based on the distance matrix; S4104, taking a region descriptor with a minimum average distance to all region descriptors of other frame images of each frame image in the region matching set as a representative region descriptor of each frame image to obtain representative region descriptors of the current observation video sequence; S4106, performing region matching based on the representative region descriptors of each frame image of the current observation video sequence and the representative region descriptors of each frame image of each initial candidate video sequence to obtain a preliminary matching result; S4108, performing weighted matching on the preliminary matching result based on confidence pyramids of frame images of the current observation video sequence and each initial candidate video sequence to obtain a spatial consistency degree between each two frame images as a weighted matching result; and S4110, obtaining a spatial consistency degree between the current observation video sequence and each initial candidate video sequence based on the spatial consistency degree between each two frame images, and obtaining multiple final candidate video sequences corresponding to the spatial consistency degree from the multiple initial candidate video sequences.

7. The method of video scene mapping and localization of claim 1, wherein, In step S500, for each of the final candidate video sequences, a co-view image cluster of each frame image in the scene map is obtained to fuse each independent final candidate video sequence into a single scene, a pose of a current observation image, i.e., a camera pose, is obtained based on feature matching results of all 3D map points of each co-view image cluster in the scene map and feature points of the current observation image, including: S5102, for each of the final candidate video sequences, frame images having a co-view relationship in the scene map and a co-view ratio greater than or equal to a preset threshold are taken out one by one to obtain a co-view image cluster having a co-view relationship greater than or equal to a co-view ratio threshold with each frame image; S5104, images in each co-view image cluster are de-duplicated to fuse each independent final candidate video sequence into a single scene; S5106, for each co-view image cluster after de-duplication, images belonging to the initial candidate video sequence are screened, and a similarity between the current observation video sequence and a global descriptor of the initial candidate video sequence to which the screened image belongs is used as a score of the screened image, and a score of an image not belonging to the initial candidate video sequence is 0, and scores of images in each co-view image cluster are accumulated to serve as a score of each co-view image cluster; and S5108, based on an order of scores of each co-view image cluster from high to low, the following steps are sequentially performed: all 3D map points of the co-view image cluster are taken out from the scene map; feature points of a last frame of the current observation video sequence, i.e., a current observation image, are extracted to perform 2D-3D feature matching with the 3D map points; and a pose of the current observation image, i.e., a camera pose, is solved based on the feature matching results and serves as an initial pose of the co-view image cluster.

8. A device for video scene mapping and positioning, characterized in that, including: a scene map establishing module, which performs: constructing an image pyramid for a current observation image of a current observation video sequence, obtaining an initial pose, a confidence pyramid and a dense feature map of the current observation image based on the image pyramid, performing iterative pose optimization in a scene map based on the initial pose, the confidence pyramid and the dense feature map of the current observation image and obtaining an optimized pose to obtain the scene map including the optimized pose of the current observation image; a video spatio-temporal feature aggregation module, which performs: performing fusion processing on depth features of all frame images of the current observation video sequence to obtain a globally-descriptive global descriptor of the video spatio-temporal feature fusion of the current observation video sequence; The spatio-temporal region matching module performs: video geolocation recognition using the global descriptor, obtaining a plurality of initial candidate video sequences with the highest similarity to the geographical location of the current observed video sequence from a pre-constructed video sequence database, performing local region feature extraction on all frame images of the current observed video sequence to obtain region descriptors of a plurality of regions of each frame image; performing region matching based on the region descriptors on each frame image of the current observed video sequence and each frame image of each initial candidate video sequence, obtaining a spatial consistency degree of each initial candidate video sequence and the current observed video sequence based on the matching result, and further obtaining a plurality of final candidate video sequences corresponding to the spatial consistency degree from the plurality of initial candidate video sequences; The multi-video geolocation recognition and scene merging module performs: for each final candidate video sequence, obtaining a co-view image cluster of each frame image in a scene map to fuse each independent final candidate video sequence into a single scene, obtaining a pose of a current observed image, i.e., a camera pose, based on feature matching results of all 3D map points of each co-view image cluster in the scene map and feature points of the current observed image; and The video geolocation positioning module performs image pyramid-based iterative pose optimization from a low-resolution image to a high-resolution image based on the pose of the current observed image to perform scene matching between the current observed image and each single scene, and obtains a geolocation positioning result of the current observed image.

9. An electronic device, comprising: Comprise: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the method for video scene mapping and positioning according to any one of claims 1 to 7.

10. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the method for video scene mapping and positioning according to any one of claims 1 to 7. The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the method for video scene mapping and positioning according to any one of claims 1 to 7.

Images