Visual map updating method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a visual map updating method and device, electronic equipment and storage medium, relates to the technical field of augmented reality, virtual reality, computer vision, deep learning and the like, and can be applied to smart city and the like. The scheme is: extracting multiple video frames from a target video; obtaining global features and local features of the multiple video frames; performing pose prediction on the multiple video frames to obtain first poses of the multiple video frames; generating a local visual map according to the first poses of the multiple video frames; and updating a visual map to be updated according to the local visual map based on the global features and the local features, to obtain a target visual map. Thus, a new local visual map is generated only according to the target video, the old visual map to be updated is updated according to the local visual map, and a complete new visual map does not need to be generated again, so that the updating efficiency of the map can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically augmented reality, virtual reality, computer vision, deep learning and other technical fields, and can be applied to smart city scenarios, especially to a visual map updating method, device, electronic device and storage medium. Background Technology

[0002] In the VPAS (Visual Positioning and Augmenting Service) task, users on the client side can use electronic devices (such as mobile phones) to take pictures of the surrounding environment and upload the pictures to the server (or server) through the client. The server performs 6-DOF positioning on a pre-established visual map, obtains the positioning results, and sends the positioning results to the client. Thus, users on the client side can experience navigation based on the positioning results.

[0003] However, some scenes in the visual map may change from time to time. If the visual map is not updated, it may be impossible to obtain the positioning results through the VPAS service, or the positioning results may be calculated incorrectly. Summary of the Invention

[0004] This disclosure provides a visual map updating method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of this disclosure, a visual map updating method is provided, comprising:

[0006] Acquire the target video and extract multiple video frames from the target video;

[0007] The global and local features corresponding to the multi-frame video are obtained, and the pose prediction of the multi-frame video is performed to obtain the first pose of the multi-frame video in the set first map coordinate system.

[0008] Based on the first pose of the multiple video frames, a local visual map in the first map coordinate system is generated;

[0009] Obtain the visual map to be updated in the set second map coordinate system, and update the visual map to be updated according to the local visual map based on the global features and the local features to obtain the target visual map.

[0010] According to another aspect of this disclosure, a visual map updating apparatus is provided, comprising:

[0011] An extraction module is used to acquire a target video and extract multiple video frames from the target video.

[0012] The prediction module is used to obtain the global and local features corresponding to the multi-frame video frames, and to perform pose prediction on the multi-frame video frames to obtain the first pose of the multi-frame video frames in the set first map coordinate system.

[0013] The generation module is used to generate a local visual map in the first map coordinate system based on the first pose of the multiple video frames.

[0014] The update module is used to obtain the visual map to be updated in the set second map coordinate system, and update the visual map to be updated according to the local visual map based on the global features and the local features to obtain the target visual map.

[0015] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the visual map update method proposed in the above aspect of this disclosure.

[0019] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided for computer instructions used to cause the computer to perform the visual map updating method proposed in the foregoing aspect of this disclosure.

[0020] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the visual map updating method proposed in the above aspect of this disclosure.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a schematic flowchart of the visual map updating method provided in Embodiment 1 of this disclosure;

[0024] Figure 2This is a schematic flowchart of the visual map updating method provided in Embodiment 2 of this disclosure;

[0025] Figure 3 This is a flowchart illustrating the visual map updating method provided in Embodiment 3 of this disclosure;

[0026] Figure 4 This is a schematic flowchart of the visual map updating method provided in Embodiment 4 of this disclosure;

[0027] Figure 5 This is a schematic flowchart of the visual map updating method provided in Embodiment 5 of this disclosure;

[0028] Figure 6 This is a schematic flowchart of the visual map updating method provided in Embodiment Six of this disclosure;

[0029] Figure 7 This is a schematic diagram of the structure of the visual map updating device provided in Embodiment 7 of this disclosure;

[0030] Figure 8 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] In related technologies, when a user discovers that a certain area cannot be located during the positioning process, the user determines the reason why the area cannot be located (e.g., the map scene has changed) and submits the reason. Professionals can then use designated camera equipment to re-capture images of the area based on the reason and recreate a visual map (or visual positioning map) using the re-captured images.

[0033] However, manually capturing images of the entire scene using a designated camera and then reconstructing the scene map from those images to generate a new visual map is computationally time-consuming. Furthermore, the resulting visual map directly replaces the original visual map instead of adding new scene elements to it, which is inefficient.

[0034] Therefore, in view of at least one of the above-mentioned problems, this disclosure proposes a visual map updating method, apparatus, electronic device and storage medium.

[0035] The visual map updating method, apparatus, electronic device, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0036] Figure 1 This is a schematic flowchart of the visual map updating method provided in Embodiment 1 of this disclosure.

[0037] The visual map update method of this disclosure can be applied to the server.

[0038] like Figure 1 As shown, the visual map update method may include the following steps:

[0039] Step 101: Obtain the target video and extract multiple video frames from the target video.

[0040] As one possible implementation, the target video can be a video actively acquired by the server. For example, the target video can be a video captured online by the server, or a video stored on the server, or a video uploaded to the server by backend staff, etc.

[0041] In this embodiment of the disclosure, after the server obtains the target video, in order to save computation and improve map update efficiency, it can extract multiple video frames from the target video frame based on video frame extraction technology.

[0042] As another possible implementation, the target video can be a video sent from the client to the server. Here, the client refers to a software program running on an electronic device that provides services to the user, such as a map application or navigation application.

[0043] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0044] In the embodiments of this disclosure, the target video can be a video collected by the client, or the target video can be a video stored in the electronic device where the client is located, or the target video can be a video collected online by the target user who logged into the client. For example, the target video can be collected online through web crawling technology, etc. This disclosure does not limit it.

[0045] In this embodiment of the disclosure, the target user who logs into the client can upload the target video to the server through the client. Correspondingly, after receiving the target video sent by the client, in order to save computation and improve map update efficiency, the server can extract multiple video frames from the target video frame based on video frame extraction technology.

[0046] Step 102: Obtain the global and local features corresponding to multiple video frames, and perform pose prediction on the multiple video frames to obtain the first pose of the multiple video frames in the set first map coordinate system.

[0047] Global features can be used to indicate the overall attributes of a video frame. For example, global features can include color features, texture features, and shape features, such as intensity histograms. Because global features are low-level visual features at the pixel level, they have good invariance, are simple to compute, and are intuitive to represent.

[0048] Local features can be features extracted from local regions within a video frame, including edges, corners, lines, curves, and regions with special attributes. Local features can be described in two main categories: corner-based and region-based. Local features are characterized by their abundance within a video frame, low correlation between features, and the fact that the disappearance of some features will not affect the detection and matching of other features even under occlusion.

[0049] In this embodiment of the disclosure, feature extraction technology can be used to extract features from multiple video frames to obtain global and local features corresponding to each video frame.

[0050] As a possible approach, in order to improve the accuracy and reliability of feature extraction, feature extraction can be performed on multiple video frames based on deep learning technology to obtain the global and local features corresponding to each video frame.

[0051] As an example, a feature extraction model may include two extraction branches: one for extracting global features of the image and the other for extracting local features of the image.

[0052] In this embodiment of the disclosure, pose prediction can also be performed on multiple video frames to obtain the first pose of each video frame in a set first map coordinate system.

[0053] As an example, the first pose is labeled as T. W1C1 , among which, T W1C1 =[R W1C1 ,t W1C1 ], R W1C1 Let t be the rotation matrix from the first map coordinate system to the camera coordinate system. W1C1 This represents the position of the camera coordinate system center in the first map coordinate system.

[0054] As one possible implementation, the first pose of each video frame in the first map coordinate system can be calculated based on the monocular SLAM (Simultaneous Localization and Mapping) algorithm.

[0055] Step 103: Generate a local visual map in the first map coordinate system based on the first pose of the multiple video frames.

[0056] In this embodiment of the disclosure, a local visual map (i.e., a new map) in a first map coordinate system can be generated based on the first pose of multiple video frames.

[0057] Step 104: Obtain the visual map to be updated in the set second map coordinate system, and update the visual map to be updated based on global and local features and the local visual map to obtain the target visual map.

[0058] In this embodiment of the disclosure, a visual map to be updated (i.e., the old map) in a second map coordinate system can be obtained. For example, the visual map to be updated can be the latest version of the visual map stored on the server side. Then, based on the global and local features of multiple video frames, the visual map to be updated can be updated according to the local visual map to obtain the target visual map.

[0059] In summary, when the target video is a video sent from the client to the server, the old visual map to be updated can be updated based on the target video uploaded by the client, without the need for professionals to carry a specific panoramic camera to a certain area to collect images and update the old visual map based on the collected images. This can reduce the image collection cycle and improve the efficiency of map updates. Furthermore, since the video is collected by the user on the client side, without the need for professionals to carry a specific camera for video collection, the degree of human intervention can be reduced and the collection cost can be saved.

[0060] The visual map update method of this disclosure extracts multiple video frames from a target video, obtains global and local features of these frames, and performs pose prediction on the frames to obtain their first pose in a first map coordinate system. Based on this first pose, a local visual map in the first map coordinate system is generated. Then, based on the global and local features, the local visual map is updated to obtain the target visual map. Therefore, on the one hand, only a new local visual map is generated from the target video, and the old visual map is updated based on this local visual map, without needing to regenerate a completely new visual map, thus improving map update efficiency. On the other hand, updating the visual map avoids situations where positioning results cannot be obtained through the VPAS service, improving the user experience.

[0061] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0062] To clearly illustrate how a local visual map in a first map coordinate system is generated based on the first pose of multiple video frames in the above embodiments, this disclosure also proposes a visual map updating method.

[0063] Figure 2 This is a schematic flowchart of the visual map updating method provided in Embodiment 2 of this disclosure.

[0064] like Figure 2 As shown, the visual map update method may include the following steps:

[0065] Step 201: Obtain the target video.

[0066] Step 202: Extract multiple video frames from the target video.

[0067] The explanation of steps 201 to 202 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0068] In a first possible implementation of any embodiment of this disclosure, in order to reduce the amount of computation and improve the efficiency of map updates, the target video can be cropped according to a set frame extraction interval to obtain multiple video frames.

[0069] The frame extraction interval is preset, for example, marked as d. For example, the first video frame, the first+d video frame, the first+2d video frame, the first+3d video frame, and so on, can be extracted from the target video.

[0070] In a second possible implementation of any embodiment of this disclosure, keyframes can be extracted from the target video to obtain multiple video frames.

[0071] As an example, keyframe extraction technology can be used to extract keyframes from a target video to obtain multiple video frames (i.e., multiple keyframes).

[0072] In a third possible implementation of any embodiment of this disclosure, considering the similarity of content between consecutive video frames, in order to reduce the amount of computation and improve processing efficiency, this disclosure can perform deduplication processing on the target video based on the similarity between the content of each video frame in the target video (referred to as the fourth similarity in this disclosure) to obtain multiple video frames.

[0073] For example, different video frames with a similarity higher than a set threshold can be deduplicated. The set threshold is pre-defined, such as 80% or 90%.

[0074] In a fourth possible implementation of any embodiment of this disclosure, considering the similarity of content between different video frames with close acquisition positions, in order to reduce the amount of computation and improve processing efficiency, this disclosure can also segment the target video according to the acquisition position corresponding to each video frame in the target video to obtain multiple video segments, and obtain multiple video frames from the multiple video segments.

[0075] For example, for each video segment, a video frame can be randomly extracted from the video segment, or the first video frame of the video segment can be extracted, or a video frame at a set position in the video segment can be extracted, etc. This disclosure does not limit this.

[0076] Therefore, it is possible to extract multiple video frames from a target video based on different methods, which can improve the flexibility and applicability of the method.

[0077] It should be noted that the above video frame extraction method is only an example, but this disclosure is not limited to it. In actual applications, other video frame extraction algorithms or video frame extraction algorithms can also be used to extract multiple video frames from the target video, and there are no restrictions on this.

[0078] Step 203: Obtain the global and local features corresponding to multiple video frames, and perform pose prediction on the multiple video frames to obtain the first pose of the multiple video frames in the set first map coordinate system.

[0079] The explanation of step 203 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0080] Step 204: Based on the first similarity between the local features of the multiple video frames, determine any two matching video frames from the multiple video frames.

[0081] In this embodiment of the disclosure, a first similarity between local features of each video frame can be calculated, and any two matching video frames can be determined from the multiple video frames based on the first similarity.

[0082] As one possible implementation, for any one of the multiple video frames, the first similarity between the local features of that video frame and the local features of the other video frames in the multiple video frames can be determined, and the target video frame with the largest first similarity can be determined from the other video frames. That video frame and the target video frame are then used as two matched video frames.

[0083] As another possible implementation, for any one of the multiple video frames, candidate video frames can be determined from the multiple video frames, wherein the difference between the shooting time of the candidate video frame and the shooting time of the current video frame is less than a set threshold. Then, a first similarity score can be calculated between the local features of each candidate video frame and the local features of the current video frame. Based on this first similarity score, a target video frame can be determined from the candidate video frames. For example, the candidate video frame with the highest first similarity score can be used as the target video frame, thus making the current video frame and the target video frame two matched video frames.

[0084] Therefore, for any given video frame, determining the target video frame that matches that video frame from only its neighboring or adjacent video frames can improve the accuracy and reliability of subsequent spatial location calculations, thereby enhancing the precision of visual map generation.

[0085] Step 205: Determine any two matching first pixels from the matched video frames.

[0086] In this embodiment of the disclosure, for any two matching video frames, any two matching first pixel points can be determined from the two matching video frames.

[0087] For example, if any two matching video frames are labeled as video frame A and video frame B, and video frame A has 500 pixels a that match pixels b in video frame B, then 500 pixel pairs (pixel a, pixel b) can be determined from video frames A and B. The two pixels in each pixel pair are the first pixels of any two matching pairs.

[0088] Step 206: Determine the first spatial position of the matched first pixel in the world coordinate system based on the first image position of the matched first pixel in the matched video frame and the first pose corresponding to the matched video frame.

[0089] In this embodiment of the disclosure, the first spatial position (i.e., 3D position) of any two matched first pixels in the world coordinate system can be determined based on the first image position (i.e., 2D position) of any two matched first pixels in the two matched video frames and the first pose corresponding to any two matched video frames. For example, the first spatial position of any two matched first pixels in the world coordinate system can be calculated based on a triangulation algorithm.

[0090] Step 207: Generate a local visual map in the first map coordinate system based on each video frame, the first pose of each video frame, and the first spatial position of the first pixel in each video frame.

[0091] In this embodiment of the disclosure, a local visual map in a first map coordinate system can be generated based on each video frame, the first pose of each video frame, and the first spatial position of the first pixel in each video frame.

[0092] Optionally, after generating the local visual map, the local visual map can be optimized based on the BA (Bundle Adjustment) optimization algorithm to make the position of each pixel in the local map and the pose of each video frame more accurate.

[0093] Step 208: Obtain the visual map to be updated in the set second map coordinate system, and update the visual map to be updated based on global and local features and the local visual map to obtain the target visual map.

[0094] The explanation of step 208 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0095] The visual map updating method of this disclosure determines any two matching video frames from multiple video frames based on a first similarity between local features; determines any two matching first pixels from the matching video frames; determines a first spatial position of the matching first pixels in the world coordinate system based on the first image position of the matching first pixels in the matching video frames and the first pose corresponding to the matching video frames; and generates a local visual map in a first map coordinate system based on each video frame, the first pose of each video frame, and the first spatial position of the first pixels in each video frame. Therefore, by simultaneously generating a visual map based on the image, the image pose, and the 3D position of each pixel in the image, the accuracy and reliability of the visual map generation result can be improved.

[0096] To clearly illustrate how the visual map to be updated is updated based on global and local features and the local visual map in any embodiment of this disclosure, this disclosure also proposes a visual map updating method.

[0097] Figure 3 This is a schematic flowchart of the visual map updating method provided in Embodiment 3 of this disclosure.

[0098] like Figure 3 As shown, the visual map update method may include the following steps:

[0099] Step 301: Obtain the target video and extract multiple video frames from the target video.

[0100] Step 302: Obtain the global and local features corresponding to multiple video frames, and perform pose prediction on the multiple video frames to obtain the first pose of the multiple video frames in the set first map coordinate system.

[0101] Step 303: Generate a local visual map in the first map coordinate system based on the first pose of the multiple video frames.

[0102] Step 304: Obtain the visual map to be updated in the set second map coordinate system.

[0103] The explanation of steps 301 to 304 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0104] Step 305: For any video frame in the local visual map, determine the matching image from the visual map to be updated based on the global and local features of the video frame.

[0105] In this embodiment of the disclosure, for any video frame in a local visual map, a matching image that matches the video frame can be determined from the visual map to be updated based on the global and local features corresponding to the video frame.

[0106] In other words, the similarity between the video frame and each visual image can be calculated based on the global and local features of the video frame, as well as the global and local features of each visual image in the visual map to be updated. Based on this similarity, a matching image is determined from the visual images that matches the video frame. For example, the visual image with the highest similarity can be used as the matching image for the video frame, or the visual image with the highest similarity and a similarity higher than a set similarity threshold can be used as the matching image for the video frame.

[0107] Step 306: Determine the second pose of the video frame in the second map coordinate system based on the matched image.

[0108] In this embodiment of the disclosure, the second pose of the aforementioned video frame in the second map coordinate system can be determined based on the matching image. For example, the second pose is labeled as T. W2C2 , among which, T W2C2 =[R W2C2 ,t W2C2 ], R W2C2 Let t be the rotation matrix from the second map coordinate system to the camera coordinate system. W2C2 This is the location of the camera coordinate system center in the second map coordinate system.

[0109] As an example, the second pose of a video frame in the second map coordinate system can be determined based on the perspective n-point PnP algorithm and the matching image.

[0110] Among them, the PnP (pespective-n-point) algorithm can include P3P, EPnP (EfficientPnP), UPnP (unifiedPnP), DLT (Direct Linear Transform), optimization solutions and other algorithms.

[0111] Step 307: Determine the mapping relationship between the first map coordinate system and the second map coordinate system based on the second pose and the first pose corresponding to each video frame.

[0112] In this embodiment of the disclosure, since the first pose is the pose of the video frame in the first map coordinate system and the second pose is the pose of the video frame in the second map coordinate system, the mapping relationship (or transformation relationship) between the first map coordinate system and the second map coordinate system can be determined based on the second pose and the first pose corresponding to each video frame.

[0113] Step 308: Based on the mapping relationship, update the visual map to be updated according to the local visual map to obtain the target visual map.

[0114] In this embodiment of the disclosure, the target visual map can be obtained by updating the visual map to be updated in the second map coordinate system based on the mapping relationship between the first map coordinate system and the second map coordinate system, according to the local visual map in the first map coordinate system.

[0115] The visual map update method of this disclosure involves determining a matching image from the visual map to be updated for any video frame in a local visual map, based on the global and local features of the video frame; determining the second pose of the video frame in a second map coordinate system based on the matching image; determining the mapping relationship between the first and second map coordinate systems based on the second and first poses corresponding to each video frame; and updating the visual map to be updated based on the mapping relationship to obtain the target visual map. Therefore, by updating the old map based on the new map according to the mapping relationship between the map coordinate systems corresponding to the old and new maps, the accuracy and reliability of the update results can be improved.

[0116] To clearly illustrate how the second pose of a video frame in the second map coordinate system is determined based on the matching image in any embodiment of this disclosure, this disclosure also proposes a visual map update method.

[0117] Figure 4 This is a schematic flowchart of the visual map updating method provided in Embodiment 4 of this disclosure.

[0118] like Figure 4As shown, the visual map update method may include the following steps:

[0119] Step 401: Obtain the target video and extract multiple video frames from the target video.

[0120] Step 402: Obtain the global and local features corresponding to multiple video frames, and perform pose prediction on the multiple video frames to obtain the first pose of the multiple video frames in the set first map coordinate system.

[0121] Step 403: Generate a local visual map in the first map coordinate system based on the first pose of the multiple video frames.

[0122] Step 404: Obtain the visual map to be updated in the set second map coordinate system.

[0123] Step 405: For any video frame in the local visual map, determine the matching image from the visual map to be updated based on the global and local features of the video frame.

[0124] The explanations of steps 401 to 405 can be found in the relevant descriptions in any embodiment of this disclosure, and will not be repeated here.

[0125] Step 406: Determine any two matching second pixel points from the matched image and video frames.

[0126] In this embodiment of the disclosure, each pixel in the matching image can be matched with each pixel in the aforementioned video frame to determine any two matching second pixel points.

[0127] For example, if 400 pixels c in the matching image match pixels d in the video frame, then 400 pixel pairs (pixel c, pixel d) can be determined from the matching image and the video frame. In each pixel pair, the two pixels are the second pixels of any two matching pairs.

[0128] Step 407: Determine the second pose of the video frame in the second map coordinate system based on the second spatial position of the matched second pixel in the visual map to be updated, and the second image position of the matched second pixel in the matched image and video frame.

[0129] In this embodiment of the disclosure, the second pose of the video frame in the second map coordinate system can be determined based on the second spatial position (i.e., 3D position) of any two matching second pixels in the visual map to be updated, and the second image position (i.e., 2D position) of the two matching second pixels in the matching image and video frame.

[0130] Using the example above, the second pose of the video frame in the second map coordinate system can be determined based on the second spatial position of pixel c in the visual map to be updated, the second image position of pixel c in the matching image, and the second image position of pixel d in the video frame.

[0131] As an example, the second pose of the video frame in the second map coordinate system can be determined by using the perspective n-point PnP algorithm based on the second spatial position of any two matched second pixels in the visual map to be updated, and the second image position of the two matched second pixels in the matched image and video frame.

[0132] Step 408: Determine the mapping relationship between the first map coordinate system and the second map coordinate system based on the second pose and the first pose corresponding to each video frame.

[0133] Step 409: Based on the mapping relationship, update the visual map to be updated according to the local visual map to obtain the target visual map.

[0134] The explanation of steps 408 to 409 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0135] The visual map updating method of this disclosure determines any two matching second pixels from matching images and video frames; and determines the second pose of the video frame in the second map coordinate system based on the second spatial position of the matching second pixels in the visual map to be updated and the second image position of the matching second pixels in the matching images and video frames. Therefore, it is possible to determine the pose of an image based on the 2D and 3D positions of each pixel in two matching images, which improves the accuracy and reliability of pose calculation.

[0136] To clearly illustrate how a matching image is determined from the visual map to be updated based on the global and local features of a video frame in any embodiment of this disclosure, this disclosure also proposes a visual map updating method.

[0137] Figure 5 This is a schematic flowchart of the visual map updating method provided in Embodiment 5 of this disclosure.

[0138] like Figure 5 As shown, the visual map update method may include the following steps:

[0139] Step 501: Obtain the target video and extract multiple video frames from the target video.

[0140] Step 502: Obtain the global and local features corresponding to multiple video frames, and perform pose prediction on the multiple video frames to obtain the first pose of the multiple video frames in the set first map coordinate system.

[0141] Step 503: Generate a local visual map in the first map coordinate system based on the first pose of the multiple video frames.

[0142] Step 504: Obtain the visual map to be updated in the set second map coordinate system.

[0143] The explanations of steps 501 to 504 can be found in the relevant descriptions in any embodiment of this disclosure, and will not be repeated here.

[0144] Step 505: For any video frame in the local visual map, determine at least one candidate image similar to the video frame from the visual map to be updated based on the global features of the video frame.

[0145] In this embodiment of the disclosure, for any video frame in a local visual map, at least one candidate image similar to the video frame can be determined from the visual map to be updated based on the global features of the video frame.

[0146] As one possible implementation, for any video frame in the local visual map, the similarity between the global features of the video frame and the global features of each visual image in the visual map to be updated (referred to as the third similarity in this disclosure) can be determined, and each visual image can be sorted from largest to smallest according to the corresponding third similarity value, so that a set number of visual images at the top of the sort can be selected as candidate images.

[0147] The number of items is set to a pre-defined threshold, such as 20, 30, 40, or 50.

[0148] As another possible implementation, for any video frame in the local visual map, the third similarity between the global features of the video frame and the global features of each visual image in the visual map to be updated can be determined, and visual images with a third similarity higher than a set similarity threshold can be used as candidate images.

[0149] Therefore, it is possible to determine candidate images from various visual images based on different methods, which can improve the flexibility and applicability of the method.

[0150] Step 506: Based on the second similarity between the local features of the video frame and the local features of each candidate image, determine the matching image that matches the video frame from each candidate image.

[0151] In this embodiment of the disclosure, a second similarity can be calculated between the local features of the video frame and the local features of each candidate image. Based on the second similarity, a matching image that matches the video frame can be determined from each candidate image. For example, the candidate image with the highest second similarity can be used as the matching image that matches the video frame.

[0152] Step 507: Determine the second pose of the video frame in the second map coordinate system based on the matched image.

[0153] Step 508: Determine the mapping relationship between the first map coordinate system and the second map coordinate system based on the second pose and the first pose corresponding to each video frame.

[0154] Step 509: Based on the mapping relationship, update the visual map to be updated according to the local visual map to obtain the target visual map.

[0155] The explanation of steps 507 to 509 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0156] The visual map update method of this disclosure, for any video frame in a local visual map, determines at least one candidate image similar to the video frame from the visual map to be updated based on the global features of the video frame; and determines a matching image from the candidate images based on a second similarity between the local features of the video frame and the local features of each candidate image. Therefore, it can simultaneously determine a matching image from each visual image in the visual map to be updated based on both the global and local features of the video frame, improving the accuracy and reliability of the determination results, thereby improving the accuracy and reliability of subsequent map updates.

[0157] To clearly illustrate how the visual map to be updated is updated based on the local visual map in any embodiment of this disclosure to obtain the target visual map, this disclosure also proposes a visual map updating method.

[0158] Figure 6 This is a schematic flowchart of the visual map updating method provided in Embodiment Six of this disclosure.

[0159] like Figure 6 As shown, the visual map update method may include the following steps:

[0160] Step 601: Obtain the target video and extract multiple video frames from the target video.

[0161] Step 602: Obtain the global and local features corresponding to multiple video frames, and perform pose prediction on the multiple video frames to obtain the first pose of the multiple video frames in the set first map coordinate system.

[0162] Step 603: Generate a local visual map in the first map coordinate system based on the first pose of the multiple video frames.

[0163] Step 604: Obtain the visual map to be updated in the set second map coordinate system.

[0164] Step 605: For any video frame in the local visual map, determine the matching image from the visual map to be updated based on the global and local features of the video frame.

[0165] Step 606: Determine the second pose of the video frame in the second map coordinate system based on the matched image.

[0166] Step 607: Determine the mapping relationship between the first map coordinate system and the second map coordinate system based on the second pose and the first pose corresponding to each video frame.

[0167] The explanation of steps 601 to 607 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0168] Step 608: Based on the mapping relationship, map the first pose of each video frame in the local visual map to the second map coordinate system to obtain the target pose of each video frame in the second map coordinate system.

[0169] In this embodiment of the disclosure, the first pose of each video frame in the local visual map under the first map coordinate system can be mapped to the second map coordinate system according to the mapping relationship between the first map coordinate system and the second map coordinate system, so as to obtain the target pose of each video frame under the second map coordinate system.

[0170] Step 609: Based on the mapping relationship, map the first spatial position of the first pixel in each video frame to the second map coordinate system to obtain the target spatial position of the first pixel in each video frame.

[0171] In this embodiment of the disclosure, the first spatial position of the first pixel in each video frame can be mapped to the second map coordinate system according to the mapping relationship between the first map coordinate system and the second map coordinate system, so as to obtain the target spatial position of the first pixel in each video frame.

[0172] Step 610: Add each video frame, the target pose corresponding to each video frame, and the target spatial position of the first pixel in each video frame to the visual map to be updated, so as to obtain the target visual map.

[0173] In this embodiment of the disclosure, video frames can be added to the visual map to be updated, and the target pose corresponding to each video frame can be added, as well as the target spatial position of the first pixel in each video frame can be added, so as to obtain the target visual map.

[0174] As an example, this invention uses a target video as the client to capture and upload it to the server for illustrative purposes. The implementation principle of any embodiment of this disclosure may include the following parts:

[0175] First, the VPAS system.

[0176] The VPAS system takes video frames or images captured by a user's handheld electronic device as input. Through steps such as image search, feature matching, and pose solving, it calculates the 6DOF (6 degrees of freedom) pose T of the camera in the user's handheld electronic device in the map coordinate system. CW , among which, T CW =[R CW ,t CW ], R CW Let t be the rotation matrix from the map coordinate system to the camera coordinate system. CW This represents the position of the camera coordinate system center within the map coordinate system.

[0177] Second, mapping of the VPAS system.

[0178] For areas that wish to utilize VPAS capabilities, professionals can carry designated panoramic cameras to acquire images of the area and upload them to the server. The server will then run a mapping algorithm to generate a visual map of the area (referred to as the visual map to be updated in this disclosure), which can then be used by the VPAS system.

[0179] Third, visual map updates based on crowdsourced data.

[0180] Step A: Create a new map using the target video uploaded by the user. new (Refered in this disclosure as a local visual map), includes the following sub-steps:

[0181] Step A1: Crop the target video to obtain multiple video frames, and calculate the location of these multiple video frames in the Map based on the SLAM algorithm. new The corresponding 6DOF pose in the first map coordinate system.

[0182] For example, a user-uploaded target video can be split into images according to a set frame rate, resulting in an image set Img_new, which includes multiple extracted video frames. Then, a monocular SLAM algorithm can be used to reconstruct each video frame from the image set onto a new map, Map. new The corresponding 6DOF pose T in the first map coordinate system W1C1 .

[0183] Step A2: Extract global and local features from the video frames.

[0184] The global feature Fg_i and the local feature Fl_i of each video frame Img_i in Img_new are extracted using a neural network.

[0185] Step A3: Feature matching, triangulation, and BA optimization.

[0186] For each video frame Img_i in Img_new, select N temporally adjacent images Img_i_target from Img_new, and use the local features extracted in step A2 to perform feature matching to obtain the matching relationship of local features on all video frames in the target video.

[0187] Based on the feature matching relationship between adjacent images, for all matching point pairs between adjacent images, since the poses of these matching points or feature points observed in step A1 are provided, the 3D position of the matching point in the world coordinate system can be calculated based on the above poses. For example, the 3D position of the matching point in the world coordinate system can be calculated based on a triangulation algorithm.

[0188] Record all successfully triangulated matching points as the new map (Map). new The map point (referred to as the first pixel in this disclosure) and the 6DOF poses of all video frames together form the initialized local visual map. new Optimize the Map using the BA algorithm. new This makes the 6DOF poses of map points and video frames more accurate. Thus, the new map is obtained. new .

[0189] Step B, Map new All video frames in the old map old (In this disclosure, it is referred to as the location in the visual map to be updated).

[0190] For the new map Map new For each video frame Img_i in the database, the global features of the video frame are used to perform image search in the old map to detect a set of images Img_i_canditate that are similar to Img_i in the old map database. The images in this set are called candidate images.

[0191] Perform local feature matching between each pair of candidate images in Img_i and Img_i_canditate to determine the matching image Img_i_match from each candidate image.

[0192] Based on the PNP algorithm, the pose of Img_i is solved using Img_i_match, and its position in the old map Map is calculated. old 6DOF pose T W2C2 .

[0193] At this point, the new map can be mapped. new The video frames are divided into two groups:

[0194] Group 1: Those successfully located in the old map, their 6DOF pose in the first map coordinate system corresponding to the new map is denoted as T. W1C1_i The 6DOF pose in the second map coordinate system corresponding to the old map is denoted as T. W2C2_i .

[0195] Group 2: Those that were not successfully located on the old map.

[0196] Step C, merge the new map. new Go to old map old middle.

[0197] Using the poses of all successfully located video frames from step B in the new and old maps, i.e., T W1C1_i and T W2C2_i The transformation relationship between the first map coordinate system corresponding to the new map and the second map coordinate system corresponding to the old map (referred to as the mapping relationship in this disclosure) is calculated. Based on the transformation relationship, the pose of each video frame in the new map is transformed to the second map coordinate system corresponding to the old map.

[0198] By utilizing the transformation relationship between the old and new map coordinate systems, the coordinate positions of all map points in the new map are transformed to the old map, and these map points are added to the old map.

[0199] In summary, allowing users to upload videos themselves, without the need for professional personnel to manually capture videos with specific cameras, reduces the level of human intervention and saves on data collection costs. Furthermore, it eliminates the need to regenerate a completely new visual map, thus improving the efficiency of visual map updates.

[0200] The visual map updating method of this disclosure maps the first pose of each video frame in a local visual map to a second map coordinate system according to a mapping relationship, to obtain the target pose of each video frame in the second map coordinate system; maps the first spatial position of the first pixel in each video frame to the second map coordinate system according to the mapping relationship, to obtain the target spatial position of the first pixel in each video frame; and adds each video frame, the corresponding target pose of each video frame, and the target spatial position of the first pixel in each video frame to the visual map to be updated, to obtain the target visual map. Therefore, not only are the poses of each image in the new map mapped to the old map, but the 3D positions of pixels in the images of the new map are also mapped to the old map, which can improve the map updating effect, i.e., improve the accuracy of map updating.

[0201] With the above Figures 1 to 6Corresponding to the visual map updating method provided in the embodiments, this disclosure also provides a visual map updating device. Since the visual map updating device provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 6 The visual map updating method provided in the embodiments corresponds to the visual map updating method provided in the embodiments of this disclosure, and therefore the implementation of the visual map updating method is also applicable to the visual map updating apparatus provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0202] Figure 7 This is a schematic diagram of the structure of the visual map updating device provided in Embodiment 7 of this disclosure.

[0203] like Figure 7 As shown, the visual map updating device 700 may include: an extraction module 701, a prediction module 702, a generation module 703, and an updating module 704.

[0204] The extraction module 701 is used to acquire the target video and extract multiple video frames from the target video.

[0205] The prediction module 702 is used to acquire global and local features corresponding to multiple video frames and perform pose prediction on the multiple video frames to obtain the first pose of the multiple video frames in the set first map coordinate system.

[0206] The generation module 703 is used to generate a local visual map in the first map coordinate system based on the first pose of multiple video frames.

[0207] The update module 704 is used to obtain the visual map to be updated in the set second map coordinate system, and update the visual map to be updated based on global and local features and the local visual map to obtain the target visual map.

[0208] In one possible implementation of this disclosure, the generation module 703 is specifically configured to: determine any two matching video frames from the multiple video frames based on a first similarity between local features of the multiple video frames; determine any two matching first pixels from the matching video frames; determine the first spatial position of the matching first pixels in the world coordinate system based on the first image position of the matching first pixels in the matching video frames and the first pose corresponding to the matching video frames; and generate a local visual map in a first map coordinate system based on each video frame, the first pose of each video frame, and the first spatial position of the first pixels in each video frame.

[0209] In one possible implementation of this disclosure, the generation module 703 is specifically configured to: for any one of the multiple video frames, determine a candidate video frame from the multiple video frames, wherein the difference between the shooting time of the candidate video frame and the shooting time of the video frame is less than a set threshold; determine a target video frame from each candidate video frame based on a first similarity between the local features of each candidate video frame and the local features of the video frame; and use the target video frame and the video frame as the matched video frames.

[0210] In one possible implementation of this disclosure, the update module 704 is specifically configured to: for any video frame in the local visual map, determine a matching image from the visual map to be updated based on the global and local features of the video frame; determine the second pose of the video frame in the second map coordinate system based on the matching image; determine the mapping relationship between the first map coordinate system and the second map coordinate system based on the second pose and the first pose corresponding to each video frame; and update the visual map to be updated based on the mapping relationship to obtain the target visual map.

[0211] In one possible implementation of this disclosure, the update module 704 is specifically configured to: determine any two matching second pixel points from the matching image and video frame; and determine the second pose of the video frame in the second map coordinate system based on the second spatial position of the matching second pixel points in the visual map to be updated and the second image position of the matching second pixel points in the matching image and video frame.

[0212] In one possible implementation of this disclosure, the update module 704 is specifically configured to: for any video frame in the local visual map, determine at least one candidate image similar to the video frame from the visual map to be updated based on the global features of the video frame; and determine a matching image that matches the video frame from each candidate image based on the second similarity between the local features of the video frame and the local features of each candidate image.

[0213] In one possible implementation of this disclosure, the update module 704 is specifically configured to: for any video frame in the local visual map, determine the third similarity between the global features of the video frame and the global features of each visual image in the visual map to be updated; sort each visual image in descending order of the corresponding third similarity value; select a set number of visual images at the top of the sort as candidate images; or, select visual images with a third similarity higher than a set similarity threshold as candidate images.

[0214] In one possible implementation of this disclosure, the update module 704 is specifically configured to: map the first pose of each video frame in the local visual map to the second map coordinate system according to the mapping relationship, so as to obtain the target pose of each video frame in the second map coordinate system; map the first spatial position of the first pixel in each video frame to the second map coordinate system according to the mapping relationship, so as to obtain the target spatial position of the first pixel in each video frame; and add each video frame, the target pose corresponding to each video frame, and the target spatial position of the first pixel in each video frame to the visual map to be updated, so as to obtain the target visual map.

[0215] In one possible implementation of this disclosure, the extraction module 701 is specifically used to: crop the target video according to a set frame extraction interval to obtain multiple video frames; or, extract keyframes from the target video to obtain multiple video frames; or, perform deduplication processing on the target video according to the fourth similarity between the content of each video frame in the target video to obtain multiple video frames; or, segment the target video according to the acquisition position corresponding to each video frame in the target video to obtain multiple video segments, and obtain multiple video frames from the multiple video segments.

[0216] The visual map updating apparatus of this disclosure extracts multiple video frames from a target video, obtains global and local features of these frames, and performs pose prediction on the frames to obtain a first pose in a first map coordinate system. Based on this first pose, a local visual map in the first map coordinate system is generated. Then, based on the global and local features, the local visual map is used to update the visual map to be updated, thus obtaining the target visual map. Therefore, by generating a new local visual map only from the target video and updating the old visual map based on this local visual map, without needing to regenerate a completely new visual map, the map updating efficiency can be improved.

[0217] To implement the above embodiments, this disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the visual map update method proposed in any of the above embodiments of this disclosure.

[0218] To implement the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the visual map update method proposed in any of the above embodiments of this disclosure.

[0219] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the visual map update method proposed in any of the above embodiments of this disclosure.

[0220] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0221] Figure 8 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0222] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 802 or a computer program loaded from storage unit 808 into RAM (Random Access Memory) 803. The RAM 803 can also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0223] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0224] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the visual map update method described above. For example, in some embodiments, the visual map update method described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the visual map update method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the visual map update method described above by any other suitable means (e.g., by means of firmware).

[0225] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0226] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0227] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0228] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0229] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0230] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0231] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0232] According to the technical solution of this disclosure, multiple video frames are extracted from a target video, and their global and local features are obtained. Pose prediction is then performed on these multiple video frames to obtain their first pose in a first map coordinate system. Based on the first pose of the multiple video frames, a local visual map in the first map coordinate system is generated. Then, based on the global and local features, the visual map to be updated is updated using the local visual map to obtain the target visual map. Therefore, only a new local visual map is generated from the target video, and the old visual map to be updated is updated based on this local visual map, without the need to regenerate a completely new visual map, thus improving map update efficiency.

[0233] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution proposed in this disclosure can be achieved, and this is not limited herein.

[0234] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A visual map updating method, comprising: obtaining a target video, and extracting a plurality of video frames from the target video; obtaining global features and local features corresponding to the plurality of video frames, and performing pose prediction on the plurality of video frames to obtain first poses of the plurality of video frames in a first map coordinate system; generating a local visual map in the first map coordinate system according to the first poses of the plurality of video frames; obtaining a to-be-updated visual map in a second map coordinate system, and for any one of the video frames in the local visual map, determining a matching image matched with the video frame from the to-be-updated visual map according to the global features and the local features of the video frame; determining a second pose of the video frame in the second map coordinate system according to the matching image; determining a mapping relationship between the first map coordinate system and the second map coordinate system according to the second poses and the first poses corresponding to the video frames; and updating the to-be-updated visual map according to the local visual map based on the mapping relationship to obtain a target visual map.

2. The method of claim 1, wherein, The generating of the local visual map in the first map coordinate system according to the first poses of the plurality of video frames comprises: determining any two matching video frames from the plurality of video frames according to first similarities between the local features of the plurality of video frames; determining any two matching first pixel points from the matching video frames; determining first spatial positions of the matching first pixel points in a world coordinate system according to first image positions of the matching first pixel points in the matching video frames and the first poses corresponding to the matching video frames; and generating the local visual map in the first map coordinate system according to the video frames, the first poses of the video frames, and the first spatial positions of the first pixel points in the video frames.

3. The method of claim 2, wherein, The determining of any two matching video frames from the plurality of video frames according to first similarities between the local features of the plurality of video frames comprises: for any one of the video frames in the plurality of video frames, determining candidate video frames from the plurality of video frames, wherein a difference between a shooting time of the candidate video frames and a shooting time of the video frame is less than a set threshold; determining target video frames from the candidate video frames according to first similarities between the local features of the candidate video frames and the local features of the video frame; and taking the target video frames and the video frame as the matching video frames.

4. The method of claim 2, wherein, The determining of a second pose of the video frame in the second map coordinate system according to the matching image comprises: determining any two matching second pixel points from the matching image and the video frame; determining the second pose of the video frame in the second map coordinate system according to second spatial positions of the matching second pixel points in the to-be-updated visual map and second image positions of the matching second pixel points in the matching image and the video frame.

5. The method of claim 2, wherein, The determining, for any one of the video frames in the local visual map, a matching image matched with the video frame from the to-be-updated visual map according to the global feature and the local feature of the video frame comprises: The determining, for any one of the video frames in the local visual map, at least one candidate image similar to the video frame from the to-be-updated visual map according to the global feature of the video frame comprises: The determining, for any one of the video frames in the local visual map, a matching image matched with the video frame from the to-be-updated visual map according to the global feature and the local feature of the video frame comprises:

6. The method of claim 5, wherein, The determining, for any one of the video frames in the local visual map, at least one candidate image similar to the video frame from the to-be-updated visual map according to the global feature of the video frame comprises: The determining, for any one of the video frames in the local visual map, a matching image matched with the video frame from the to-be-updated visual map according to the global feature and the local feature of the video frame comprises: The determining, for any one of the video frames in the local visual map, a matching image matched with the video frame from the to-be-updated visual map according to the global feature and the local feature of the video frame comprises: The determining, for any one of the video frames in the local visual map, at least one candidate image similar to the video frame from the to-be-updated visual map according to the global feature of the video frame comprises: The determining, for any one of the video frames in the local visual map, a matching image matched with the video frame from the to-be-updated visual map according to the global feature and the local feature of the video frame comprises:

7. The method of claim 2, wherein, The determining, for any one of the video frames in the local visual map, a matching image matched with the video frame from the to-be-updated visual map according to the global feature and the local feature of the video frame comprises: The determining, for any one of the video frames in the local visual map, a matching image matched with the video frame from the to-be-updated visual map according to the global feature and the local feature of the video frame comprises: The updating, based on the mapping relationship, the to-be-updated visual map according to the local visual map to obtain a target visual map comprises: The mapping, according to the mapping relationship, the first pose of each of the video frames in the local visual map to the second map coordinate system to obtain a target pose of each of the video frames in the second map coordinate system; 8. The method of any one of claims 1-7, wherein, The mapping, according to the mapping relationship, the first spatial position of the first pixel point in each of the video frames to the second map coordinate system to obtain a target spatial position of the first pixel point in each of the video frames; The adding, in the to-be-updated visual map, each of the video frames, the target pose corresponding to each of the video frames, and the target spatial position of the first pixel point in each of the video frames to obtain the target visual map. The extracting, from the target video, a plurality of video frames comprises: The performing, according to a set frame extraction interval, a cropping processing on the target video to obtain the plurality of video frames; Or, The performing, on the target video, a key frame extraction to obtain the plurality of video frames; Or, The performing, according to a fourth similarity between contents of each frame of the target video, a deduplication processing on the target video to obtain the plurality of video frames; Or, The performing, according to a collection position corresponding to each frame of the target video, a segmentation processing on the target video to obtain a plurality of video segments, and obtaining the plurality of video frames from the plurality of video segments.

9. A visual map updating apparatus, comprising: an extraction module configured to obtain a target video and extract a plurality of video frames from the target video; The prediction module is configured to acquire global features and local features corresponding to the multiple video frames, and perform pose prediction on the multiple video frames to obtain first poses of the multiple video frames in a first map coordinate system. The generation module is configured to generate a local visual map in the first map coordinate system according to the first poses of the multiple video frames. The update module is configured to acquire a to-be-updated visual map in a second map coordinate system, and update the to-be-updated visual map according to the local visual map based on the global features and the local features to obtain a target visual map. The update module is specifically configured to: For any one of the video frames in the local visual map, determine a matching image matched with the video frame from the to-be-updated visual map according to the global features and the local features of the video frame. Determine a second pose of the video frame in the second map coordinate system according to the matching image. Determine a mapping relationship between the first map coordinate system and the second map coordinate system according to the second poses and the first poses of the video frames. Update the to-be-updated visual map according to the local visual map based on the mapping relationship to obtain a target visual map.

10. The apparatus of claim 9, wherein, The generation module is specifically configured to: Determine any two matching video frames from the multiple video frames according to a first similarity between the local features of the multiple video frames. Determine any two matching first pixel points from the matching video frames. Determine a first spatial position of the matching first pixel points in a world coordinate system according to first image positions of the matching first pixel points in the matching video frames and the first poses of the matching video frames. Generate the local visual map in the first map coordinate system according to the video frames, the first poses of the video frames, and the first spatial positions of the first pixel points in the video frames.

11. The apparatus of claim 10, wherein, The generation module is specifically configured to: For any one of the video frames in the multiple video frames, determine a candidate video frame from the multiple video frames, where a difference between a shooting time of the candidate video frame and a shooting time of the video frame is less than a set threshold. Determine a target video frame from the candidate video frames according to a first similarity between the local features of the candidate video frames and the local features of the video frame. Take the target video frame and the video frame as the matching video frames.

12. The apparatus of claim 10, wherein, The update module is specifically configured to: Determine any two matching second pixel points from the matching image and the video frame. Determine a second pose of the video frame in the second map coordinate system according to a second spatial position of the matching second pixel points in the to-be-updated visual map and second image positions of the matching second pixel points in the matching image and the video frame.

13. The apparatus of claim 10, wherein, The update module is specifically configured to: For any one of the video frames in the local visual map, determine at least one candidate image similar to the video frame from the to-be-updated visual map according to the global features of the video frame. According to a second similarity between the local feature of the video frame and the local feature of each of the candidate images, the matching image matching the video frame is determined from each of the candidate images.

14. The apparatus of claim 13, wherein, The updating module is specifically configured to: For any one of the video frames in the local visual map, determine a third similarity between a global feature of the video frame and a global feature of each visual image in the visual map to be updated; Sort each of the visual images according to the value of the corresponding third similarity from large to small; Select a set number of the visual images in the front as the candidate images; or Select a visual image whose third similarity is higher than a set similarity threshold as the candidate image.

15. The apparatus of claim 10, wherein, The updating module is specifically configured to: According to the mapping relationship, map the first pose of each of the video frames in the local visual map to the second map coordinate system to obtain a target pose of each of the video frames in the second map coordinate system; According to the mapping relationship, map the first spatial position of the first pixel point in each of the video frames to the second map coordinate system to obtain a target spatial position of the first pixel point in each of the video frames; Add each of the video frames, the corresponding target pose of each of the video frames, and the target spatial position of the first pixel point in each of the video frames in the visual map to be updated to obtain the target visual map.

16. The apparatus of any one of claims 9-15, wherein, The extraction module is specifically configured to: According to a set frame extraction interval, perform cropping processing on the target video to obtain the plurality of video frames; Or, Perform key frame extraction on the target video to obtain the plurality of video frames; Or, According to a fourth similarity between the contents of each frame of the target video, perform deduplication processing on the target video to obtain the plurality of video frames; Or, According to the collection positions corresponding to each frame of the target video, perform segmentation processing on the target video to obtain a plurality of video segments, and obtain the plurality of video frames from the plurality of video segments.

17. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images