Visual positioning method and apparatus, device, and medium

Abstract

The present disclosure provides a visual positioning method and apparatus, a device, and a storage medium. The method comprises: when a multi-ocular camera collects a plurality of current images at a current moment, determining first global descriptors respectively corresponding to the plurality of current images; stitching the first global descriptors respectively corresponding to the plurality of current images, to determine a first global stitched descriptor of the multi-ocular camera at the current moment; acquiring a visual sparse map corresponding to a target scene, wherein the visual sparse map comprises second global stitched descriptors respectively corresponding to the multi-ocular camera at different historical moments; on the basis of the first global stitched descriptor and the second global stitched descriptors respectively corresponding to different historical moments, determining, from a plurality of historical images respectively collected by the multi-ocular camera at different historical moments, a plurality of historical target images matching the plurality of current images; and on the basis of the plurality of current images and the plurality of historical target images, determining scene pose data of a local scene in the target scene. High speed, low costs and high precision are realized.

Description

Visual positioning method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese invention patent application number 202310871351.9, entitled “Visual positioning method, device, equipment and medium” and filed on July 14, 2023, and the entire application is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of visual positioning technology, and in particular to a visual positioning method, device, equipment and medium. Background Art

[0004] Visual positioning technology is required for positioning in many fields, including virtual reality (VR), autonomous driving, and robot navigation. For example, when a user wears VR glasses, the position of the local scene they see in the real world needs to be located in real time. Based on the position of the local scene in the real world, the local scene seen by the user is rendered, allowing the user to view a rendered image of the local scene.

[0005] Current visual positioning methods generally use a monocular camera to capture scene images and perform mapping based on these images to obtain a constructed scene image. Keyframe extraction and feature matching are then performed on this constructed scene image, and pose calculation is performed using Perspective-n-Point (PnP) technology to achieve visual positioning. However, the positioning method based on a monocular camera has low accuracy and cannot meet the requirements of high-precision visual positioning. Therefore, proposing a high-precision visual positioning method is a technical problem that urgently needs to be solved.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a visual positioning method, device, equipment and medium.

[0008] In a first aspect, the present disclosure provides a visual positioning method, the method comprising:

[0009] When the multi-camera captures a plurality of current images at a current moment, determining first global descriptors corresponding to the plurality of current images respectively, wherein the scenes corresponding to the plurality of current images are local scenes in the target scene;

[0010] Stitching the first global descriptors corresponding to the multiple current images respectively to determine the first global stitching descriptor of the multi-camera at the current moment;

[0011] Obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-constructed from a plurality of historical images captured by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors corresponding to the multi-camera at different historical moments;

[0012] Determining, based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively captured by the multi-camera at the different historical moments;

[0013] Based on the multiple current images and the multiple historical target images, scene pose data of the local scene in the target scene is determined.

[0014] In a second aspect, the present disclosure provides a visual positioning device, the device comprising:

[0015] A first determination module is configured to determine, when the multi-camera captures a plurality of current images at a current moment, first global descriptors corresponding to the plurality of current images, respectively, wherein the scenes corresponding to the plurality of current images are local scenes in the target scene;

[0016] a stitching module, configured to stitch the first global descriptors corresponding to the multiple current images respectively, and determine the first global stitching descriptor of the multi-camera at the current moment;

[0017] an acquisition module, configured to acquire a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-constructed from a plurality of historical images captured by the multi-camera at different historical moments, and the visual sparse map includes a second global splicing descriptor corresponding to each of the multi-cameras at different historical moments;

[0018] a second determining module, configured to determine, based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively captured by the multi-camera at the different historical moments;

[0019] A positioning module is used to determine the scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

[0020] In a third aspect, the present disclosure provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a terminal device, the terminal device implements the above method.

[0021] In a fourth aspect, the present disclosure provides a device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0022] In a fifth aspect, the present disclosure provides a computer program product, which includes a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor.

[0023] The technical solution provided by the embodiments of the present disclosure has at least the following advantages compared with the prior art:

[0024] The embodiments of the present disclosure provide a visual positioning method, apparatus, device and medium, the method comprising: when a multi-eye camera captures multiple current images at a current moment, determining first global descriptors corresponding to the multiple current images respectively, wherein the scenes corresponding to the multiple current images are local scenes in a target scene; splicing the first global descriptors corresponding to the multiple current images respectively, and determining the first global splicing descriptor of the multi-eye camera at the current moment; obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-constructed by mapping multiple historical images respectively captured by the multi-eye camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-eye camera at different historical moments; based on the first global splicing descriptor and the second global splicing descriptors respectively corresponding to different historical moments, determining multiple historical target images matching the multiple current images from the multiple historical images respectively captured by the multi-eye camera at different historical moments; and determining scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images. Through the above method, the first global stitching descriptor of the multi-camera at the current moment and the second global stitching descriptors corresponding to the multi-camera at different historical moments can be used to determine multiple historical target images and multiple current images with matching relationships for visual positioning. In this way, visual positioning based on the multi-camera is achieved quickly and at low cost, while improving the accuracy of visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] FIG1 is a schematic diagram of a flow chart of a visual positioning method provided by an embodiment of the present disclosure;

[0028] FIG2 is a logic diagram of a visual positioning method provided by an embodiment of the present disclosure;

[0029] FIG3 is a schematic structural diagram of a visual positioning device provided by an embodiment of the present disclosure;

[0030] FIG4 is a schematic structural diagram of a visual positioning device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0033] Figure 1 shows a flow chart of a visual positioning method provided by an embodiment of the present disclosure. As shown in Figure 1, the visual positioning method includes the following steps.

[0034] S110. When the multi-camera captures multiple current images at a current moment, determine first global descriptors corresponding to the multiple current images, respectively, wherein the scenes corresponding to the multiple current images are local scenes in the target scene.

[0035] In this embodiment, in scenarios such as users wearing VR glasses for a virtual reality experience, autonomous vehicles performing autonomous driving, and robot navigation, it is necessary to determine the position data of the local scene in which the device itself is located within the global scene. Specifically, multiple cameras are used to capture images of the local scene in which the device is located in real time, which are used as the current image. The images captured by each camera correspond to the same timestamp. The position data of the local scene in which the device is located in the global scene is then calculated based on the multiple current images to achieve visual positioning of the device.

[0036] In this embodiment, visual positioning is achieved using Simultaneous Localization and Mapping (SLAM) technology. SLAM includes a mapping process and a positioning process. Specifically, in the mapping process, a camera is first used to capture an image of the global scene, and then the image of the global scene is processed to construct an offline image of the global scene. In the positioning process, a camera is first used to capture an image of the local scene in real time. Then, based on the coordinate data of the feature points in the image of the local scene and the coordinate data of the feature points in the offline image of the global scene, the pose data of the local scene in the global scene is determined, and the pose data is used as the positioning result of the local scene, thereby achieving a visual positioning effect.

[0037] The number of multi-cameras must be at least 2. The multi-cameras can be installed in front of, on top of, or on the left and right sides of VR glasses, autonomous vehicles, robots, and other devices, without limitation.

[0038] The target scene refers to the global scene in which the device is located, while the local scene refers to the portion of the scene within the field of view of the multi-camera. For example, when a user wears a VR device for a virtual reality experience, the target scene is the entire living room, and the local scene is the TV wall or the roof within the living room. For another example, when an autonomous vehicle is operating autonomously, the global scene is the road scene in XX City, and the local scene is the road scene on YY Street.

[0039] Furthermore, after acquiring the multiple current images, descriptor calculations are performed on the multiple current images respectively to obtain first global descriptors corresponding to the multiple current images respectively.

[0040] In the embodiment of the present disclosure, the method for determining the first global descriptor includes but is not limited to the following methods: extracting feature points of each current image to obtain two-dimensional feature points of each current image; based on the two-dimensional feature points of each current image, calculating the descriptor of each current image to obtain the first global descriptor corresponding to each current image.

[0041] Specifically, a preset feature point detection algorithm is used to extract feature points of each current image to obtain two-dimensional feature points of each current image, and then the neighborhood area of ​​each feature point is divided into blocks, the gradient histogram of each block is calculated, and the first global descriptor corresponding to each current image is calculated based on the gradient histogram.

[0042] Wherein, the descriptor can be a vector. Optionally, the first global descriptor corresponding to the current image captured by each camera is denoted as I p , and the first global descriptor corresponding to each current image is a 1*n vector.

[0043] Optionally, the preset feature point detection algorithm may include but is not limited to an ORB feature point detection algorithm, a pattern recognition algorithm, and the like.

[0044] In other embodiments, a first global descriptor capable of characterizing each current image may be directly determined without detecting the two-dimensional feature points of each current image.

[0045] S120: Splice the first global descriptors corresponding to the multiple current images to determine the first global splicing descriptor of the multi-camera at the current moment.

[0046] In order to improve the accuracy of visual positioning, after determining the first global descriptors corresponding to multiple current images, these first global descriptors are spliced. The first global splicing descriptors of the multi-camera at the current moment are used to facilitate the subsequent matching of multiple current images with multiple historical images at different historical moments based on the first global splicing descriptors.

[0047] In this embodiment, optionally, S120 specifically includes: splicing the first global descriptors corresponding to multiple current images according to the camera identifier of each camera to obtain a first candidate splicing descriptor of the multi-camera at the current moment; normalizing the first candidate splicing descriptor to obtain a first global splicing descriptor.

[0048] The camera identifier is used to represent the position or serial number of each camera. Optionally, based on the camera identifier, the first global descriptors corresponding to the multiple current images are spliced ​​row by row, and after normalization, a first global spliced ​​descriptor set is formed. The first global spliced ​​descriptor set is denoted as G1(I p ), if p is 4, the first global concatenated descriptor set is recorded as G1(I1, I2, I3, I4), that is, G1(I1, I2, I3, I4) is a 1*4n vector.

[0049] S130. Obtain a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-constructed using multiple historical images captured by multiple cameras at different historical moments, and the visual sparse map includes second global splicing descriptors corresponding to the multiple cameras at different historical moments.

[0050] In this embodiment, before locating the local scene where the device is located, the multi-camera is used to collect multiple historical images of the target scene at different times, and the multiple historical images are processed to generate a visual sparse map corresponding to the target scene, and then the visual sparse map is used to locate the local scene where the device is located.

[0051] In this embodiment, the method of generating a visual sparse map includes but is not limited to the following methods: obtaining multiple historical images captured by multi-cameras at the same time; extracting feature points of the multiple historical images respectively to obtain two-dimensional feature points of the multiple historical images respectively; based on the two-dimensional feature points corresponding to the multiple historical images respectively, performing descriptor calculations on the multiple historical images respectively to obtain second global descriptors corresponding to the multiple historical images respectively; based on the second global descriptors corresponding to the multiple historical images respectively, generating a visual sparse map including the second global splicing descriptors corresponding to the multi-cameras at different historical moments.

[0052] In other embodiments, a second global descriptor capable of characterizing each historical image may be directly determined without detecting the two-dimensional feature points of each historical image.

[0053] Among them, according to the second global descriptors corresponding to multiple historical images, a visual sparse map containing the second global stitching descriptors corresponding to the multi-camera at different historical moments is generated, including: for the multiple historical images collected at each historical moment, according to the camera identifier of each camera, the second global descriptors corresponding to the multiple historical images collected at each historical moment are stitched together to obtain the second candidate stitching descriptors of the multi-camera at each historical moment; and the second candidate stitching descriptors corresponding to the multi-camera at different historical moments are normalized to obtain the second global stitching descriptor.

[0054] Optionally, the second global descriptor corresponding to the historical image collected by each camera is denoted as I q , and the second global descriptor corresponding to each historical image is a 1*n vector. Based on the camera identifier, the second global descriptors corresponding to multiple historical images collected at each historical moment are spliced ​​and normalized row by row to form a second global spliced ​​descriptor set. The second global spliced ​​descriptor set is recorded as G2(I q ), if q is 4, the second global concatenated descriptor set is recorded as G2(I1, I2, I3, I4), that is, G2(I1, I2, I3, I4) is a 1*4n vector.

[0055] In this embodiment, the visual sparse map may also include the two-dimensional feature points of each historical image and their corresponding three-dimensional feature points. Specifically, the method for determining the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image includes: first, determining the local descriptor of the two-dimensional feature points in each historical image, then, based on the local descriptor of the two-dimensional feature points and the second global descriptor corresponding to each historical image, projecting the two-dimensional feature points into the global scene to find the corresponding three-dimensional feature points, then, performing triangulation calculation and feature point optimization processing on the two-dimensional feature points and their corresponding three-dimensional feature points in any two historical images, thereby optimizing the position data of the three-dimensional feature points in the global scene, and using the optimized three-dimensional feature points as the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image.

[0056] Among them, the method for determining the local descriptor of the two-dimensional feature point in each historical image includes: dividing the neighborhood area of ​​the two-dimensional feature point in each historical image into blocks, calculating the gradient histogram of each block, and calculating the local descriptor of the two-dimensional feature point in each historical image based on the gradient histogram.

[0057] S140: Based on the first global stitching descriptor and the second global stitching descriptors corresponding to different historical moments, determine multiple historical target images that match the multiple current images from multiple historical images captured by the multi-camera at different historical moments.

[0058] In this embodiment, after obtaining the first global stitching descriptor and the second global stitching descriptors corresponding to different historical moments, multiple current images are matched with multiple historical images collected at different historical moments based on the first global stitching descriptor and the second global stitching descriptor, so as to find multiple historical images collected at a certain historical moment that match the multiple current images from the multiple historical images as multiple historical target images.

[0059] In order to accurately screen out multiple historical target images, S140 specifically includes: calculating the dot product of the first global splicing descriptor and the second global splicing descriptors corresponding to different historical moments; obtaining the second global splicing descriptor that generates the maximum dot product from the second global splicing descriptors corresponding to different historical moments; and using the multiple historical images corresponding to the second global splicing descriptor that generates the maximum dot product as multiple historical target images.

[0060] The dot product is the product of the length of a vector and its projection onto another vector, and it reflects the degree of "match" between two vectors. In other words, the larger the dot product, the higher the degree of match between the multiple historical images representing a certain historical moment and the multiple current images; conversely, the smaller the dot product, the lower the degree of match between the multiple historical images representing a certain historical moment and the multiple current images.

[0061] Therefore, using dot product as a parameter to measure the degree of matching, multiple historical target images that best match multiple current images are found from multiple historical images collected at multiple historical moments, which facilitates visual positioning based on multiple historical target images and helps improve the visual positioning effect.

[0062] S150 : Determine scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

[0063] In this embodiment, the current image captured by each camera is matched with the historical target image, and based on the matching result, the scene pose data of the local scene in the target scene is determined, and the scene pose data is used as the positioning result of the local scene.

[0064] Therefore, by performing visual positioning based on multiple historical target images that have the highest matching degree with multiple current images, the positioning accuracy of the local scene is improved.

[0065] An embodiment of the present disclosure provides a visual positioning method, which includes: when a multi-eye camera captures multiple current images at a current moment, determining first global descriptors corresponding to the multiple current images respectively, wherein the scenes corresponding to the multiple current images are local scenes in a target scene; splicing the first global descriptors corresponding to the multiple current images respectively, and determining the first global splicing descriptor of the multi-eye camera at the current moment; obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-constructed by mapping multiple historical images respectively captured by the multi-eye camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-eye camera at different historical moments; based on the first global splicing descriptor and the second global splicing descriptors respectively corresponding to different historical moments, determining multiple historical target images that match the multiple current images from the multiple historical images respectively captured by the multi-eye camera at different historical moments; and determining scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images. Through the above method, the first global stitching descriptor of the multi-camera at the current moment and the second global stitching descriptors corresponding to the multi-camera at different historical moments can be used to determine multiple historical target images and multiple current images with matching relationships for visual positioning. In this way, visual positioning based on the multi-camera is achieved quickly and at low cost, while improving the accuracy of visual positioning.

[0066] In another embodiment of the present disclosure, for multiple current images and multiple historical target images, the matching point pairs of each camera are first determined, and then based on the matching point pairs of each camera, an optimization iteration method is used to determine the scene pose data of the local scene in the target scene.

[0067] In the embodiment of the present disclosure, optionally, S150 specifically includes:

[0068] S1501, matching the two-dimensional feature points in the current image captured by each camera with the three-dimensional feature points in the historical target image captured by each camera to obtain matching point pairs for each camera;

[0069] S1502: Determine the scene pose data of the local scene in the target scene based on the matching point pairs of each camera.

[0070] Among them, S1501 specifically includes: based on the first local descriptor of the two-dimensional feature points in the current image captured by each camera and the second local descriptor of the two-dimensional feature points in the historical target images captured by each camera, matching the two-dimensional feature points in the current image captured by each camera with the two-dimensional feature points in the historical target images captured by each camera to determine the two-dimensional feature point pairs; based on the two-dimensional feature point pairs and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target images, determining the matching point pairs.

[0071] The method for determining the first local descriptor of a two-dimensional feature point in the current image includes: dividing the neighborhood of the two-dimensional feature point in the current image into blocks, calculating the gradient histogram of each block, and calculating the first local descriptor based on the gradient histogram. Similarly, the method for determining the second local descriptor of a two-dimensional feature point in the historical target image captured by each camera includes: dividing the neighborhood of the two-dimensional feature point in the historical target image captured by each camera into blocks, calculating the gradient histogram of each block, and calculating the second local descriptor based on the gradient histogram.

[0072] Therefore, the images of each camera can be matched based on the local descriptors to accurately determine the matching point pairs of each camera.

[0073] In order to avoid reducing the accuracy of feature point matching due to camera shake or other factors, the multi-camera collects multiple continuous image frame sequences at the current moment, and the multiple current images are the current image frame sequences in the multiple continuous image frame sequences. Accordingly, S130 specifically includes: matching the two-dimensional feature points of the multiple current images in the current image frame sequence collected by each camera with the three-dimensional feature points in the historical target images collected by each camera, determining multiple first image pairs for each camera, and determining the position of each feature point in each first image pair; for each image sequence frame other than the current image frame sequence in the multiple continuous image frame sequences, based on the relative position relationship between each image sequence frame and the current image frame sequence and the position of each feature point in the first image pair, determining multiple second image pairs for each camera, wherein the multiple first image pairs and the multiple second image pairs constitute the image pairs of each camera.

[0074] The relative position relationship may be the relative posture between adjacent image frames.

[0075] Thus, when multiple consecutive image frame sequences are acquired, multiple first image pairs are determined for each camera based on any one of the image frame sequences. Subsequently, multiple second image pairs are determined for each camera based on the relative positions between adjacent image frame sequences and the positions of the feature points in the matching point pairs corresponding to each camera. This eliminates the need to perform dot product calculations on all image frame sequences, improving the matching efficiency of the two types of images acquired by each camera. Furthermore, feature point matching is performed on both the two-dimensional and three-dimensional feature points in all image pairs for each camera, thereby improving feature matching accuracy. This avoids deviations in the feature point matching process caused by camera shake or other factors, and contributes to improved visual positioning accuracy.

[0076] Among them, S1502 specifically includes: using a preset error equation to calculate the error of the first pose data corresponding to the two-dimensional feature points in multiple matching point pairs and the second pose data corresponding to the three-dimensional feature points to obtain the current error parameter; iteratively adjusting the current error value until the current error value meets the iteration stop condition, obtaining the target transformation pose between the two-dimensional feature points and the three-dimensional feature points in multiple matching point pairs, and using the target transformation pose as the scene pose data of the local scene in the target scene.

[0077] During the positioning process, when the multi-camera is started to capture images, the current positioning coordinate system of the multi-camera is determined, and the first pose data corresponding to the 2D feature points is determined in the current positioning coordinate system. During the mapping process, when the multi-camera is started to capture images, the historical mapping coordinate system is determined, and the second pose data corresponding to the 3D feature points is determined in the historical mapping coordinate system. Optionally, both the current positioning coordinate system and the historical mapping coordinate system can be map coordinate systems.

[0078] Specifically, the first pose data corresponding to the two-dimensional feature point and the second pose data corresponding to the three-dimensional feature point in the matching point pair of each camera are used as input data of the preset error equation, so that the current error value is calculated using the preset error equation, and then the current error value is iteratively adjusted until the current error value is equal to a specific threshold or tends to be stable. It is determined that the current error value meets the iteration stop condition, and the target transformed pose is obtained.

[0079] Optionally, the current error value may be a reprojection error value or an error value in another form. The specific threshold may be 0.

[0080] Optionally, the preset error equation may be expressed as follows:

[0081] Among them, cost is the current error value, which can be the reprojection error value, M is the number of images collected by the multi-camera, N M is the number of matching point pairs between any current image and historical image that have a matching relationship. π(*) is a mapping function that projects the feature points in the image coordinate system onto the pixels of the image using the camera intrinsic parameters. and It is the first pose data of the two-dimensional feature point in the matching point pair in the current positioning coordinate system. It is the second pose data of the three-dimensional feature point in the matching point pair in the historical mapping coordinate system. is the pixel coordinate of the 3D feature point in the matching point pair in the current image where the 2D feature point in the matching point pair is located, t wm and R wm is the target transformed pose.

[0082] Therefore, for the first pose data corresponding to the two-dimensional feature points and the second pose data corresponding to the three-dimensional feature points in multiple matching point pairs of each camera, the preset error equation can be used and all the first pose data and second pose data can be processed in an iterative manner to obtain the scene pose data of the local scene in the target scene, so as to achieve accurate positioning of the local scene.

[0083] In other cases, for the first camera in the multi-camera, a preset error equation can be used to determine the first transformation pose, and then, based on the external parameters of the first camera, the external parameters of at least one other camera, and the first transformation pose, the second transformation pose corresponding to at least one other camera is calculated. Finally, the first transformation pose and the second transformation pose corresponding to at least one other camera are fused to obtain the scene pose data of the local scene in the target scene.

[0084] In summary, when visual positioning is required, the matching point pairs of each camera are first determined. Then, based on the matching point pairs of each camera, an optimized iterative method is used to determine the scene pose data of the local scene in the target scene. This improves the efficiency and accuracy of visual positioning.

[0085] In another embodiment of the present disclosure, a visual localization method is explained in its entirety. The specific visual localization method includes a mapping process and a localization process. For ease of understanding, FIG2 shows a logical diagram of the visual localization method.

[0086] As shown in Figure 2, the mapping process of the visual positioning method includes:

[0087] S1. Obtain multiple historical images collected by multiple cameras at different historical moments.

[0088] Specifically, a multi-camera is used to capture images of a target scene at different historical moments to obtain multiple historical images.

[0089] S2. Generate a visual sparse map based on multiple historical images.

[0090] In some cases, first, a multi-camera is used to capture images of the target scene at different historical moments to obtain multiple historical images. Then, feature points are extracted from each historical image to obtain two-dimensional feature points of each historical image. Based on the two-dimensional feature points of each historical image, a descriptor is calculated for each historical image to obtain a second global descriptor and a second local descriptor corresponding to each historical image. According to the second global descriptors corresponding to the multiple historical images captured at the same historical moment, the second global splicing descriptors corresponding to the multi-camera at different historical moments are calculated. At the same time, according to the second global descriptor and two-dimensional feature points corresponding to each historical image, the multiple historical images are triangulated and feature points are optimized to determine the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image. Finally, a visual sparse map is generated based on the two-dimensional feature points of each historical image, the second global descriptor and second local descriptor corresponding to each historical image, the second global splicing descriptors corresponding to the multi-camera at different historical moments, and the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image.

[0091] The second global splicing descriptor is generated, specifically including: for multiple historical images collected at each historical moment, according to the camera identifier of each camera, splicing the second global descriptors corresponding to the multiple historical images collected at each historical moment to obtain the second candidate splicing descriptor of the multi-camera at each historical moment; and normalizing the second candidate splicing descriptors corresponding to the multi-camera at different historical moments to obtain the second global splicing descriptor.

[0092] Continuing with FIG2 , the positioning process of the visual positioning method includes:

[0093] S3. Obtain multiple current images captured by the multi-camera at the current moment.

[0094] Specifically, a multi-camera is used to capture images of a local scene in a target scene at a current moment to obtain a plurality of current images. Optionally, the plurality of current images may be a current image frame sequence in a plurality of continuous image frame sequences, or may be images in a single image frame sequence.

[0095] S4. Determine first global descriptors corresponding to the multiple current images respectively.

[0096] Specifically, feature points are extracted for each current image to obtain two-dimensional feature points of each current image, and descriptors are calculated for each current image based on the two-dimensional feature points of each current image to obtain a first global descriptor corresponding to each current image.

[0097] S5. Stitching the first global descriptors corresponding to the multiple current images to determine the first global stitching descriptor of the multi-camera at the current moment.

[0098] Specifically, according to the camera identifier of each camera, the first global descriptors corresponding to multiple current images are spliced ​​to obtain a first candidate spliced ​​descriptor of the multi-camera at the current moment; the first candidate spliced ​​descriptors are normalized to obtain a first global spliced ​​descriptor.

[0099] S6. Based on the first global stitching descriptor and the second global stitching descriptors corresponding to different historical moments, determine multiple historical target images that match the multiple current images from the multiple historical images respectively captured by the multi-camera at the different historical moments.

[0100] Specifically, the dot product of the first global splicing descriptor and the second global splicing descriptors corresponding to different historical moments is calculated; the second global splicing descriptor that generates the maximum dot product is obtained from the second global splicing descriptors corresponding to different historical moments; and the multiple historical images corresponding to the second global splicing descriptors that generate the maximum dot product are used as multiple historical target images.

[0101] S7. Determine scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

[0102] Specifically, the two-dimensional feature points in the current image captured by each camera are matched with the three-dimensional feature points in the historical target image captured by each camera to obtain matching point pairs for each camera; based on the matching point pairs for each camera, the scene pose data of the local scene in the target scene is determined.

[0103] The two-dimensional feature points in the current image captured by each camera are matched with the three-dimensional feature points in the historical target images captured by each camera to obtain matching point pairs for each camera, including: based on the first local descriptor of the two-dimensional feature points in the current image captured by each camera and the second local descriptor of the two-dimensional feature points in the historical target images captured by each camera, the two-dimensional feature points in the current image captured by each camera are matched with the two-dimensional feature points in the historical target images captured by each camera to determine the two-dimensional feature point pairs; based on the two-dimensional feature point pairs and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target images, the matching point pairs are determined.

[0104] Among them, based on the matching point pairs of each camera, the scene pose data of the local scene in the target scene is determined, including: using a preset error equation to calculate the error of the first pose data corresponding to the two-dimensional feature points in multiple matching point pairs and the second pose data corresponding to the three-dimensional feature points to obtain a current error value; iteratively adjusting the current error value until the current error value meets the iteration stop condition, obtaining the target transformation pose between the two-dimensional feature points and the three-dimensional feature points in the multiple matching point pairs, and using the target transformation pose as the scene pose data of the local scene in the target scene.

[0105] Based on the same inventive concept as the above method embodiment, the present disclosure further provides a visual positioning device. Referring to FIG3 , which is a schematic structural diagram of a visual positioning device provided in an embodiment of the present disclosure, the visual positioning device 300 includes:

[0106] A first determining module 310 is configured to determine, when the multi-camera captures multiple current images at a current moment, first global descriptors corresponding to the multiple current images, wherein the scenes corresponding to the multiple current images are local scenes in the target scene;

[0107] A stitching module 320 is configured to stitch the first global descriptors corresponding to the multiple current images to determine a first global stitching descriptor of the multi-camera at the current moment;

[0108] an acquisition module 330 for acquiring a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-constructed from a plurality of historical images captured by the multi-camera at different historical moments, and the visual sparse map includes a second global stitching descriptor corresponding to each of the multi-cameras at different historical moments;

[0109] A second determining module 340 is configured to determine, based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, a plurality of historical target images that match the plurality of current images from the plurality of historical images respectively captured by the multi-camera at the different historical moments;

[0110] The positioning module 350 is configured to determine the scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

[0111] In an optional embodiment, the splicing module 320 includes:

[0112] a stitching unit, configured to stitch together the first global descriptors corresponding to the multiple current images according to the camera identifier of each camera, to obtain a first candidate stitching descriptor of the multi-camera at the current moment;

[0113] A normalization unit is used to perform normalization processing on the first candidate splicing descriptor to obtain the first global splicing descriptor.

[0114] In an optional embodiment, the visual sparse map further includes second global descriptors corresponding to the plurality of historical images respectively; the device further includes:

[0115] A first determining module is configured to, for each of the plurality of historical images acquired at each historical moment, stitch together the second global descriptors corresponding to the plurality of historical images acquired at the respective historical moments according to the camera identifier of each camera, to obtain a second candidate stitching descriptor of the multi-camera at the respective historical moments;

[0116] The second determination module is used to normalize the second candidate splicing descriptors corresponding to the multi-camera at the different historical moments to obtain the second global splicing descriptor.

[0117] In an optional implementation, the second determining module 340 includes:

[0118] a calculation unit, configured to calculate a dot product of the first global splicing descriptor and the second global splicing descriptors corresponding to the different historical moments;

[0119] A first acquisition unit is configured to acquire, from the second global splicing descriptors corresponding to the different historical moments, a second global splicing descriptor that generates a maximum dot product;

[0120] The second acquisition unit is configured to use the multiple historical images corresponding to the second global splicing descriptor that generates the maximum dot product as the multiple historical target images.

[0121] In an optional implementation, the positioning module 350 includes:

[0122] a matching point pair determination unit, configured to match the two-dimensional feature points in the current image captured by each camera with the three-dimensional feature points in the historical target image captured by each camera, to obtain matching point pairs for each camera;

[0123] A positioning unit is used to determine the scene pose data of the local scene in the target scene based on the matching point pairs of each camera.

[0124] In an optional implementation manner, the matching point pair determination unit is specifically configured to:

[0125] Based on the first local descriptor of the two-dimensional feature points in the current image captured by each camera and the second local descriptor of the two-dimensional feature points in the historical target images captured by each camera, the two-dimensional feature points in the current image captured by each camera are matched with the two-dimensional feature points in the historical target images captured by each camera to determine a two-dimensional feature point pair;

[0126] The matching point pair is determined based on the two-dimensional feature point pair and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target image.

[0127] In an optional implementation manner, the positioning unit is specifically configured to:

[0128] Performing error calculation on the first pose data corresponding to the two-dimensional feature points and the second pose data corresponding to the three-dimensional feature points in the plurality of matching point pairs using a preset error equation to obtain a current error value;

[0129] The current error value is iteratively adjusted until the current error value meets the iteration stop condition, and the target transformation pose between the two-dimensional feature points and the three-dimensional feature points in the multiple matching point pairs is obtained, and the target transformation pose is used as the scene pose data of the local scene in the target scene.

[0130] An embodiment of the present disclosure provides a visual positioning device, which, when a multi-eye camera captures multiple current images at a current moment, determines first global descriptors corresponding to the multiple current images respectively, wherein the scenes corresponding to the multiple current images are local scenes in a target scene; splices the first global descriptors corresponding to the multiple current images respectively, to determine the first global splicing descriptor of the multi-eye camera at the current moment; obtains a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-constructed by mapping multiple historical images respectively captured by the multi-eye camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-eye camera at different historical moments; based on the first global splicing descriptor and the second global splicing descriptors respectively corresponding to different historical moments, determines multiple historical target images matching the multiple current images from the multiple historical images respectively captured by the multi-eye camera at different historical moments; and determines scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images. Through the above method, the first global stitching descriptor of the multi-camera at the current moment and the second global stitching descriptors corresponding to the multi-camera at different historical moments can be used to determine multiple historical target images and multiple current images with matching relationships for visual positioning. In this way, visual positioning based on the multi-camera is achieved quickly and at low cost, while improving the accuracy of visual positioning.

[0131] In addition to the above-mentioned method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device implements the visual positioning method of the embodiments of the present disclosure.

[0132] The embodiments of the present disclosure further provide a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the visual positioning method of the embodiments of the present disclosure is implemented.

[0133] In addition, the embodiment of the present disclosure further provides a visual positioning device, as shown in FIG4 , the visual positioning device may include:

[0134] Processor 401, memory 402, input device 403, and output device 404. The visual positioning device may include one or more processors 401, with one processor being used as an example in FIG4 . In some embodiments of the present disclosure, processor 401, memory 402, input device 403, and output device 404 may be connected via a bus or other means, with FIG4 using a bus as an example.

[0135] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing of the visual positioning device by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory 402 can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The input device 403 can be used to receive input digital or character information, and to generate signal input related to the observation user settings and function control of the visual positioning device.

[0136] Specifically in this embodiment, the processor 401 will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the applications stored in the memory 402, thereby realizing the various functions of the above-mentioned visual positioning device.

[0137] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0138] The above are merely specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to these embodiments, but is to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A visual positioning method, the method comprising: When the multi-eye camera collects multiple current images at the current moment, determine first global descriptors corresponding to the multiple current images respectively, wherein the scenes corresponding to the multiple current images are local scenes in the target scene; Splicing the first global descriptors respectively corresponding to the multiple current images to determine the first global splicing descriptor of the multi-eye camera at the current moment; Obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped by mapping a plurality of historical images respectively collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-camera at different historical moments; Based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, determining a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments; Based on the multiple current images and the multiple historical target images, scene pose data of the local scene in the target scene is determined.

2. The method according to claim 1, wherein: The step of stitching the first global descriptors respectively corresponding to the multiple current images to determine the first global stitching descriptor of the multi-eye camera at the current moment includes: According to the camera identifier of each eye camera, splicing the first global descriptors respectively corresponding to the multiple current images to obtain a first candidate splicing descriptor of the multi-eye camera at the current moment; The first candidate splicing descriptor is normalized to obtain the first global splicing descriptor.

3. The method according to claim 1, wherein: The visual sparse map further includes second global descriptors corresponding to the plurality of historical images respectively; the method further includes: For multiple historical images collected at each historical moment, according to the camera Identification, splicing the second global descriptors corresponding to the multiple historical images collected at each historical moment, to obtain the second candidate splicing descriptor of the multi-eye camera at each historical moment; Normalize the second candidate splicing descriptors corresponding to the multi-camera at the different historical moments to obtain the second global splicing descriptor.

4. The method according to claim 1, wherein: The determining, based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments, comprises: Calculating the dot product of the first global splicing descriptor and the second global splicing descriptors corresponding to the different historical moments; Obtaining a second global concatenated descriptor that generates a maximum dot product from the second global concatenated descriptors corresponding to the different historical moments; The multiple historical images corresponding to the second global splicing descriptor that generates the maximum dot product are used as the multiple historical target images.

5. The method according to claim 1, wherein: The determining, based on the multiple current images and the multiple historical target images, scene pose data of the local scene in the target scene comprises: Matching the two-dimensional feature points in the current image collected by each camera with the three-dimensional feature points in the historical target image collected by each camera to obtain matching point pairs of each camera; Based on the matching point pairs of each camera, the scene pose data of the local scene in the target scene is determined.

6. The method according to claim 5, wherein: The method of matching the two-dimensional feature points in the current image collected by each camera with the three-dimensional feature points in the historical target image collected by each camera to obtain the matching point pairs of each camera includes: Based on the first local descriptor of the two-dimensional feature points in the current image collected by each camera and the second local descriptor of the two-dimensional feature points in the historical target image collected by each camera, the two-dimensional feature points in the current image collected by each camera are compared with the two-dimensional feature points in the historical target image collected by each camera. Match the two-dimensional feature points in the historical target image to determine the two-dimensional feature point pairs; The matching point pair is determined based on the two-dimensional feature point pair and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target image.

7. The method according to claim 5, wherein: The determining, based on the matching point pairs of each camera, the scene pose data of the local scene in the target scene comprises: Using a preset error equation, an error calculation is performed on the first pose data corresponding to the two-dimensional feature points and the second pose data corresponding to the three-dimensional feature points in the plurality of matching point pairs to obtain a current error value; The current error value is iteratively adjusted until the current error value satisfies an iteration stop condition, a target transformation pose between the two-dimensional feature points and the three-dimensional feature points in the multiple matching point pairs is obtained, and the target transformation pose is used as the scene pose data of the local scene in the target scene.

8. A visual positioning device, comprising: A first determination module is used to determine first global descriptors corresponding to the multiple current images respectively when the multi-eye camera collects multiple current images at the current moment, wherein the scenes corresponding to the multiple current images are local scenes in the target scene; A stitching module, used to stitch the first global descriptors respectively corresponding to the multiple current images, and determine the first global stitching descriptor of the multi-eye camera at the current moment; An acquisition module is used to acquire a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped by a plurality of historical images respectively collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-camera at different historical moments; A second determination module is used to determine, based on the first global splicing descriptor and the second global splicing descriptors corresponding to the different historical moments, a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments; A positioning module is used to determine the scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

9. A computer-readable storage medium storing instructions, which, when executed on a terminal device, enable the terminal device to implement the method according to any one of claims 1 to 7.

10. A device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images