Visual positioning method and device, equipment and medium

Abstract

The invention provides a visual positioning method and device, equipment and a storage medium, and the method comprises the steps: determining first global descriptors corresponding to a plurality of current images when a multi-view camera collects the current images at a current moment; splicing the first global descriptors respectively corresponding to the plurality of current images, and determining a first global spliced descriptor of the multi-view camera at the current moment; obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map comprises second global splicing descriptors corresponding to the multi-view camera at different historical moments; based on the first global splicing descriptor and second global splicing descriptors respectively corresponding to different historical moments, determining a plurality of historical target images matched with the plurality of current images from a plurality of historical images respectively collected by a multi-view camera at different historical moments; the scene pose data of the local scene in the target scene is determined based on the multiple current images and the multiple historical target images, the speed is high, the cost is low, and the precision is high.

Description

Technical Field

[0001] The present disclosure relates to the field of visual positioning technology, and in particular to a visual positioning method, device, equipment and medium. Background Art

[0002] Visual positioning technology is required for positioning in many fields such as virtual reality (VR), autonomous driving, and robot navigation. For example, when a user wears VR glasses, it is necessary to locate the position of the local scene seen by the user in the real scene in real time, and render the local scene seen by the user based on the position of the local scene in the real scene, so that the user can see the rendered image of the local scene.

[0003] The current visual positioning method generally uses a monocular camera to capture scene images, and then builds a map based on the scene images captured by the monocular camera to obtain a constructed scene image, and then extracts key frames and matches features of the constructed scene image, and combines the perspective-n-point (PnP) technology to calculate the pose, thereby achieving visual positioning. However, the positioning method based on the monocular camera has low accuracy and cannot meet the needs of high-precision visual positioning. Therefore, proposing a high-precision visual positioning method is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a visual positioning method, device, equipment and medium.

[0005] In a first aspect, the present disclosure provides a visual positioning method, the method comprising:

[0006] When the multi-eye camera collects multiple current images at the current moment, determine first global descriptors corresponding to the multiple current images respectively, wherein the scenes corresponding to the multiple current images are local scenes in the target scene;

[0007] Splicing the first global descriptors respectively corresponding to the multiple current images to determine the first global splicing descriptor of the multi-eye camera at the current moment;

[0008] Obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped by mapping a plurality of historical images respectively collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-camera at different historical moments;

[0009] Based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, determining a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments;

[0010] Based on the multiple current images and the multiple historical target images, scene pose data of the local scene in the target scene is determined.

[0011] In a second aspect, the present disclosure provides a visual positioning device, the device comprising:

[0012] A first determination module is used to determine first global descriptors corresponding to the multiple current images respectively when the multi-eye camera collects multiple current images at the current moment, wherein the scenes corresponding to the multiple current images are local scenes in the target scene;

[0013] A stitching module, used to stitch the first global descriptors respectively corresponding to the multiple current images, and determine the first global stitching descriptor of the multi-eye camera at the current moment;

[0014] An acquisition module is used to acquire a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped by a plurality of historical images respectively collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-camera at different historical moments;

[0015] A second determination module is used to determine, based on the first global splicing descriptor and the second global splicing descriptors corresponding to the different historical moments, a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments;

[0016] A positioning module is used to determine the scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

[0017] In a third aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the above method.

[0018] In a fourth aspect, the present disclosure provides a device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0019] In a fifth aspect, the present disclosure provides a computer program product, wherein the computer program product comprises a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor.

[0020] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:

[0021] The disclosed embodiments provide a visual positioning method, apparatus, device and medium, the method comprising: when a multi-eye camera collects multiple current images at a current moment, determining first global descriptors respectively corresponding to the multiple current images, wherein the scenes corresponding to the multiple current images are local scenes in a target scene; splicing the first global descriptors respectively corresponding to the multiple current images to determine the first global splicing descriptor of the multi-eye camera at the current moment; obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped to multiple historical images respectively collected by the multi-eye camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-eye camera at different historical moments; based on the first global splicing descriptor and the second global splicing descriptors respectively corresponding to different historical moments, determining multiple historical target images matching the multiple current images from the multiple historical images respectively collected by the multi-eye camera at different historical moments; based on the multiple current images and the multiple historical target images, determining scene pose data of the local scene in the target scene. Through the above method, the first global stitching descriptor of the multi-eye camera at the current moment and the second global stitching descriptor corresponding to the multi-eye camera at different historical moments can be used to determine multiple historical target images and multiple current images with matching relationships for visual positioning. In this way, the visual positioning based on the multi-eye camera is realized quickly and at low cost, and the accuracy of the visual positioning is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 A flowchart of a visual positioning method provided by an embodiment of the present disclosure;

[0025] Figure 2 A logic diagram of a visual positioning method provided by an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of the structure of a visual positioning device provided in an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of the structure of a visual positioning device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] Figure 1 FIG. 2 shows a flow chart of a visual positioning method provided by an embodiment of the present disclosure. Figure 1 As shown, the visual positioning method includes the following steps.

[0031] S110. When the multi-eye camera captures a plurality of current images at a current moment, determine first global descriptors corresponding to the plurality of current images respectively, wherein the scenes corresponding to the plurality of current images are local scenes in the target scene.

[0032] In this embodiment, in scenarios such as users wearing VR glasses for virtual reality experience, autonomous driving vehicles for autonomous driving, and robot navigation, it is necessary to determine the position data of the local scene where the device is located in the global scene. Specifically, a multi-camera is used to collect images of the local scene where the device is located in real time as the current image, so that the images collected by each camera correspond to the same timestamp, and then the position data of the local scene where the device is located in the global scene is calculated based on multiple current images to achieve visual positioning of the device.

[0033] In this embodiment, the simultaneous localization and mapping (SLAM) technology is used to achieve visual positioning, and SLAM includes a mapping process and a positioning process. Specifically, in the mapping process, the image of the global scene is first collected by the camera, and then the image of the global scene is processed to build a global scene offline image; in the positioning process, the image of the local scene is first collected by the camera in real time, and then the position and posture data of the local scene in the global scene are determined based on the coordinate data of the feature points in the image of the local scene and the coordinate data of the feature points in the offline image of the global scene, and the position and posture data are used as the positioning result of the local scene, so as to achieve the visual positioning effect.

[0034] The number of multi-eye cameras is at least 2. The multi-eye cameras are installed in front of, on top of, or on the left and right sides of VR glasses, autonomous driving vehicles, robots, and other equipment without limitation.

[0035] The target scene refers to the global scene where the device is located, and the local scene refers to the part of the scene within the field of view of the multi-camera. For example, when a user wears a VR device for a virtual reality experience, the target scene is the entire living room, and the local scene is the TV wall or the roof of the living room. For another example, when an autonomous driving vehicle is performing autonomous driving, the global scene is the road scene of XX City, and the local scene is the road scene of YY Street.

[0036] Furthermore, after acquiring the multiple current images, descriptor calculations are performed on the multiple current images respectively to obtain first global descriptors corresponding to the multiple current images respectively.

[0037] In the embodiments of the present disclosure, the method for determining the first global descriptor includes but is not limited to the following methods: extracting feature points of each current image to obtain two-dimensional feature points of each current image; based on the two-dimensional feature points of each current image, calculating the descriptor of each current image to obtain the first global descriptor corresponding to each current image.

[0038] Specifically, a preset feature point detection algorithm is used to extract feature points of each current image to obtain two-dimensional feature points of each current image, and then the neighborhood area of ​​each feature point is divided into blocks, the gradient histogram of each block is calculated, and the first global descriptor corresponding to each current image is calculated based on the gradient histogram.

[0039] Wherein, the descriptor can be a vector. Optionally, the first global descriptor corresponding to the current image captured by each camera is denoted as I p , and the first global descriptor corresponding to each current image is a 1*n vector.

[0040] Optionally, the preset feature point detection algorithm may include but is not limited to an ORB feature point detection algorithm, a pattern recognition algorithm, and the like.

[0041] In other embodiments, a first global descriptor capable of characterizing each current image may be directly determined without detecting the two-dimensional feature points of each current image.

[0042] S120: Splice the first global descriptors corresponding to the multiple current images to determine the first global splicing descriptor of the multi-eye camera at the current moment.

[0043] In order to improve the accuracy of visual positioning, after determining the first global descriptors corresponding to multiple current images respectively, these first global descriptors are spliced ​​to use the first global splicing descriptors of the multi-eye camera at the current moment, so as to facilitate the subsequent matching of multiple current images with multiple historical images at different historical moments based on the first global splicing descriptors.

[0044] In this embodiment, optionally, S120 specifically includes: according to the camera identifier of each eye camera, splicing the first global descriptors corresponding to multiple current images respectively to obtain a first candidate splicing descriptor of the multi-eye camera at the current moment; normalizing the first candidate splicing descriptor to obtain a first global splicing descriptor.

[0045] The camera identifier is used to represent the position or serial number of each camera. Optionally, based on the camera identifier, the first global descriptors corresponding to the multiple current images are spliced ​​row by row, and after normalization, a first global spliced ​​descriptor set is formed, and the first global spliced ​​descriptor set is denoted as G1(I p ), if p is 4, the first global concatenated descriptor set is recorded as G1(I1, I2, I3, I4), that is, G1(I1, I2, I3, I4) is a 1*4n vector.

[0046] S130. Obtain a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped using a plurality of historical images collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors corresponding to the multi-camera at different historical moments.

[0047] In this embodiment, before locating the local scene where the device is located, the multi-eye camera is used to collect multiple historical images of the target scene at different times, and the multiple historical images are processed to generate a visual sparse map corresponding to the target scene, and then the visual sparse map is used to locate the local scene where the device is located.

[0048] In this embodiment, the method of generating a visual sparse map includes but is not limited to the following methods: obtaining multiple historical images collected by a multi-eye camera at the same time; extracting feature points of the multiple historical images respectively to obtain two-dimensional feature points of the multiple historical images respectively; based on the two-dimensional feature points corresponding to the multiple historical images respectively, performing descriptor calculations on the multiple historical images respectively to obtain second global descriptors corresponding to the multiple historical images respectively; based on the second global descriptors corresponding to the multiple historical images respectively, generating a visual sparse map containing the second global splicing descriptors corresponding to the multi-eye cameras at different historical moments.

[0049] In other embodiments, a second global descriptor capable of characterizing each historical image may be directly determined without detecting the two-dimensional feature points of each historical image.

[0050] Among them, according to the second global descriptors corresponding to the multiple historical images respectively, a visual sparse map including the second global stitching descriptors corresponding to the multi-eye cameras at different historical moments is generated, including: for the multiple historical images collected at each historical moment, according to the camera identification of each eye camera, the second global descriptors corresponding to the multiple historical images collected at each historical moment are stitched to obtain the second candidate stitching descriptors of the multi-eye camera at each historical moment; the second candidate stitching descriptors corresponding to the multi-eye cameras at different historical moments are normalized to obtain the second global stitching descriptor.

[0051] Optionally, the second global descriptor corresponding to the historical image collected by each camera is denoted as I q , and the second global descriptor corresponding to each historical image is a 1*n vector. Based on the camera identifier, the second global descriptors corresponding to multiple historical images collected at each historical moment are spliced ​​and normalized by row to form a second global spliced ​​descriptor set. The second global spliced ​​descriptor set is recorded as G2(I q ), if q is 4, the second global concatenated descriptor set is recorded as G2(I1, I2, I3, I4), that is, G2(I1, I2, I3, I4) is a 1*4n vector.

[0052] In this embodiment, the visual sparse map may also include the two-dimensional feature points of each historical image and their corresponding three-dimensional feature points. Specifically, the method for determining the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image includes: first, determining the local descriptor of the two-dimensional feature points in each historical image, then, based on the local descriptor of the two-dimensional feature points and the second global descriptor corresponding to each historical image, projecting the two-dimensional feature points into the global scene to find the corresponding three-dimensional feature points, then, performing triangulation calculation and feature point optimization processing on the two-dimensional feature points and their corresponding three-dimensional feature points in any two historical images, thereby optimizing the position data of the three-dimensional feature points in the global scene, and using the optimized three-dimensional feature points as the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image.

[0053] Among them, the method for determining the local descriptor of the two-dimensional feature point in each historical image includes: dividing the neighborhood area of ​​the two-dimensional feature point in each historical image into blocks, calculating the gradient histogram of each block, and calculating the local descriptor of the two-dimensional feature point in each historical image based on the gradient histogram.

[0054] S140. Based on the first global stitching descriptor and the second global stitching descriptors corresponding to different historical moments, determine a plurality of historical target images matching the plurality of current images from a plurality of historical images respectively captured by the multi-eye camera at different historical moments.

[0055] In this embodiment, after obtaining the first global stitching descriptor and the second global stitching descriptor corresponding to different historical moments, multiple current images are matched with multiple historical images collected at different historical moments based on the first global stitching descriptor and the second global stitching descriptor, so as to find multiple historical images collected at a certain historical moment that match the multiple current images from the multiple historical images as multiple historical target images.

[0056] In order to accurately screen out multiple historical target images, S140 specifically includes: calculating the dot product of the first global splicing descriptor and the second global splicing descriptors corresponding to different historical moments; obtaining the second global splicing descriptor that generates the maximum dot product from the second global splicing descriptors corresponding to different historical moments; and using the multiple historical images corresponding to the second global splicing descriptor that generates the maximum dot product as multiple historical target images.

[0057] The dot product refers to the product of the length of a vector and its projection on another vector, and the dot product can reflect the "matching degree" of two vectors. That is, if the dot product is larger, the matching degree between the multiple historical images representing a certain historical moment and the multiple current images is higher; on the contrary, if the dot product is smaller, the matching degree between the multiple historical images representing a certain historical moment and the multiple current images is lower.

[0058] Therefore, using the dot product as a parameter to measure the degree of matching, multiple historical target images that best match multiple current images are found from multiple historical images collected at multiple historical moments, which facilitates visual positioning based on multiple historical target images and helps improve the visual positioning effect.

[0059] S150: Determine scene pose data of a local scene in a target scene based on a plurality of current images and a plurality of historical target images.

[0060] In this embodiment, the current image captured by each camera is matched with the historical target image, and based on the matching result, the scene pose data of the local scene in the target scene is determined, and the scene pose data is used as the positioning result of the local scene.

[0061] Therefore, by performing visual positioning based on multiple historical target images with the highest matching degree with multiple current images, the positioning accuracy of the local scene is improved.

[0062] The disclosed embodiment provides a visual positioning method, which includes: when a multi-eye camera captures multiple current images at a current moment, determining first global descriptors corresponding to the multiple current images respectively, wherein the scenes corresponding to the multiple current images are local scenes in a target scene; splicing the first global descriptors corresponding to the multiple current images respectively, and determining the first global splicing descriptor of the multi-eye camera at the current moment; obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped to multiple historical images respectively captured by the multi-eye camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-eye camera at different historical moments; based on the first global splicing descriptor and the second global splicing descriptors respectively corresponding to different historical moments, determining multiple historical target images matching the multiple current images from the multiple historical images respectively captured by the multi-eye camera at different historical moments; and determining scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images. Through the above method, the first global stitching descriptor of the multi-eye camera at the current moment and the second global stitching descriptor corresponding to the multi-eye camera at different historical moments can be used to determine multiple historical target images and multiple current images with matching relationships for visual positioning. In this way, the visual positioning based on the multi-eye camera is realized quickly and at low cost, and the accuracy of the visual positioning is improved.

[0063] In another embodiment of the present disclosure, for multiple current images and multiple historical target images, the matching point pairs of each camera are first determined, and then based on the matching point pairs of each camera, an optimization iteration method is used to determine the scene pose data of the local scene in the target scene.

[0064] In the embodiment of the present disclosure, optionally, S150 specifically includes:

[0065] S1501, matching the two-dimensional feature points in the current image collected by each camera with the three-dimensional feature points in the historical target image collected by each camera to obtain matching point pairs of each camera;

[0066] S1502: Determine the scene pose data of the local scene in the target scene based on the matching point pairs of each camera.

[0067] Among them, S1501 specifically includes: based on the first local descriptor of the two-dimensional feature points in the current image collected by each camera and the second local descriptor of the two-dimensional feature points in the historical target images collected by each camera, the two-dimensional feature points in the current image collected by each camera are matched with the two-dimensional feature points in the historical target images collected by each camera to determine the two-dimensional feature point pairs; based on the two-dimensional feature point pairs and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target images, the matching point pairs are determined.

[0068] Among them, the method for determining the first local descriptor of the two-dimensional feature point in the current image includes: performing regional block division on the neighborhood area of ​​the two-dimensional feature point in the current image, calculating the gradient histogram of each block, and calculating the first local descriptor based on the gradient histogram. Similarly, the method for determining the second local descriptor of the two-dimensional feature point in the historical target image collected by each camera includes: performing regional block division on the neighborhood area of ​​the two-dimensional feature point in the historical target image collected by each camera, calculating the gradient histogram of each block, and calculating the second local descriptor based on the gradient histogram.

[0069] Therefore, the images of each camera can be matched based on the local descriptor to accurately determine the matching point pairs of each camera.

[0070] In order to avoid reducing the accuracy of feature point matching due to camera shake or other factors, the multi-camera collects multiple continuous image frame sequences at the current moment, and the multiple current images are the current image frame sequences in the multiple continuous image frame sequences. Accordingly, S130 specifically includes: matching the two-dimensional feature points of the multiple current images in the current image frame sequence collected by each camera with the three-dimensional feature points in the historical target image collected by each camera, determining multiple first image pairs of each camera, and determining the position of each feature point in each first image pair; for each image sequence frame other than the current image frame sequence in the multiple continuous image frame sequences, based on the relative position relationship between each other image sequence frame and the current image frame sequence and the position of each feature point in the first image pair, determining multiple second image pairs of each camera, wherein the multiple first image pairs and the multiple second image pairs constitute the image pairs of each camera.

[0071] The relative position relationship may be the relative position and posture between adjacent image frames.

[0072] Therefore, when a plurality of continuous image frame sequences are collected, a plurality of first image pairs of each camera are determined based on any image frame sequence, and then a plurality of second image pairs of each camera are determined based on the relative positions between adjacent image frame sequences and the positions of each feature point in the matching point pair corresponding to each camera. Therefore, it is not necessary to perform dot product calculations on all image frame sequences, which improves the matching efficiency of the two types of images collected by each camera, and further performs feature point matching on the two-dimensional feature points and three-dimensional feature points in all image pairs of each camera, thereby improving the feature matching accuracy, thereby avoiding deviations in the feature point matching process due to camera shake or other factors, and is conducive to improving the visual positioning accuracy.

[0073] Among them, S1502 specifically includes: using a preset error equation to calculate the error of the first pose data corresponding to the two-dimensional feature points in multiple matching point pairs and the second pose data corresponding to the three-dimensional feature points to obtain a current error parameter; iteratively adjusting the current error value until the current error value meets the iteration stop condition, obtaining the target transformation pose between the two-dimensional feature points and the three-dimensional feature points in multiple matching point pairs, and using the target transformation pose as the scene pose data of the local scene in the target scene.

[0074] During the positioning process, when the multi-eye camera is started to collect images, the current positioning coordinate system of the multi-eye camera at the current moment is determined, and the first pose data corresponding to the two-dimensional feature point is determined in the current positioning coordinate system. During the mapping process, when the multi-eye camera is started to collect images, the historical mapping coordinate system is determined, and the second pose data corresponding to the three-dimensional feature point is determined in the historical mapping coordinate system. Optionally, both the current positioning coordinate system and the historical mapping coordinate system can be map coordinate systems (Map).

[0075] Specifically, the first pose data corresponding to the two-dimensional feature point and the second pose data corresponding to the three-dimensional feature point in the matching point pair of each camera are used as input data of the preset error equation, so that the current error value is calculated using the preset error equation, and then the current error value is iteratively adjusted until the current error value is equal to a specific threshold or tends to be stable. It is determined that the current error value meets the iteration stop condition, and the target transformed pose is obtained.

[0076] Optionally, the current error value may be a reprojection error value or an error value in another form. The specific threshold may be 0.

[0077] Optionally, the preset error equation may be expressed as follows:

[0078]

[0079] Among them, cost is the current error value, which can be the reprojection error value, M is the number of images collected by the multi-camera, and N M is the number of matching point pairs between any current image and historical image that have a matching relationship. π(*) is a mapping function that projects the feature points in the image coordinate system onto the pixels of the image using the camera intrinsic parameters. and It is the first position data of the two-dimensional feature point in the matching point pair in the current positioning coordinate system. It is the second pose data of the 3D feature point in the matching point pair in the historical mapping coordinate system. is the pixel coordinate of the 3D feature point in the matching point pair in the current image where the 2D feature point in the matching point pair is located, t wm and R wm is the target transformed pose.

[0080] Therefore, for the first pose data corresponding to the two-dimensional feature points and the second pose data corresponding to the three-dimensional feature points in multiple matching point pairs of each eye camera, the preset error equation can be used to process all the first pose data and the second pose data in an iterative manner to obtain the scene pose data of the local scene in the target scene, so as to achieve accurate positioning of the local scene.

[0081] In other cases, for the first camera in the multi-camera, a preset error equation can be used to determine the first transformation pose, and then, based on the extrinsic parameters of the first camera, the extrinsic parameters of at least one other camera, and the first transformation pose, the second transformation pose corresponding to at least one other camera is calculated. Finally, the first transformation pose and the second transformation pose corresponding to at least one other camera are fused to obtain the scene pose data of the local scene in the target scene.

[0082] In summary, when visual positioning is required, the matching point pairs of each camera are first determined, and then based on the matching point pairs of each camera, an optimized iterative method is used to determine the scene pose data of the local scene in the target scene. This improves the efficiency and accuracy of visual positioning.

[0083] In another embodiment of the present disclosure, the visual positioning method is explained as a whole. The specific visual positioning method includes a mapping process and a positioning process. For ease of understanding, Figure 2 A schematic diagram showing the logic of the vision positioning method.

[0084] like Figure 2 As shown in Figure 1, the mapping process of the visual positioning method includes:

[0085] S1. Obtain multiple historical images collected by multi-cameras at different historical moments.

[0086] Specifically, a multi-camera is used to collect images of the target scene at different historical moments to obtain multiple historical images.

[0087] S2. Generate visual sparse maps based on multiple historical images.

[0088] In some cases, first, a multi-eye camera is used to collect images of a target scene at different historical moments to obtain multiple historical images. Then, feature points are extracted from each historical image to obtain two-dimensional feature points of each historical image. Based on the two-dimensional feature points of each historical image, a descriptor is calculated for each historical image to obtain a second global descriptor and a second local descriptor corresponding to each historical image. According to the second global descriptors corresponding to multiple historical images collected at the same historical moment, the second global splicing descriptors corresponding to the multi-eye camera at different historical moments are calculated. At the same time, according to the second global descriptor and two-dimensional feature points corresponding to each historical image, the multiple historical images are triangulated and feature points are optimized to determine the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image. Finally, a visual sparse map is generated based on the two-dimensional feature points of each historical image, the second global descriptor and second local descriptor corresponding to each historical image, the second global splicing descriptors corresponding to the multi-eye camera at different historical moments, and the three-dimensional feature points corresponding to the two-dimensional feature points of each historical image.

[0089] Among them, generating the second global stitching descriptor specifically includes: for multiple historical images collected at each historical moment, according to the camera identifier of each camera, stitching the second global descriptors corresponding to the multiple historical images collected at each historical moment to obtain the second candidate stitching descriptor of the multi-camera at each historical moment; normalizing the second candidate stitching descriptors corresponding to the multi-camera at different historical moments to obtain the second global stitching descriptor.

[0090] Continue to see Figure 2 ,The positioning process of the visual positioning method includes:

[0091] S3. Acquire multiple current images captured by the multi-eye camera at the current moment.

[0092] Specifically, a multi-eye camera is used to collect images of a local scene in a target scene at a current moment to obtain a plurality of current images. Optionally, the plurality of current images may be a current image frame sequence in a plurality of continuous image frame sequences, or may be images in an image frame sequence.

[0093] S4. Determine first global descriptors corresponding to multiple current images respectively.

[0094] Specifically, feature points are extracted for each current image to obtain two-dimensional feature points of each current image, and descriptor calculation is performed on each current image based on the two-dimensional feature points of each current image to obtain a first global descriptor corresponding to each current image.

[0095] S5. Splice the first global descriptors corresponding to the multiple current images to determine the first global splicing descriptor of the multi-eye camera at the current moment.

[0096] Specifically, according to the camera identifier of each eye camera, the first global descriptors corresponding to the multiple current images are spliced ​​to obtain the first candidate splicing descriptor of the multi-eye camera at the current moment; the first candidate splicing descriptor is normalized to obtain the first global splicing descriptor.

[0097] S6. Based on the first global stitching descriptor and the second global stitching descriptors corresponding to different historical moments, determine multiple historical target images matching the multiple current images from multiple historical images respectively captured by the multi-eye camera at the different historical moments.

[0098] Specifically, the dot product of the first global splicing descriptor and the second global splicing descriptors corresponding to different historical moments is calculated; the second global splicing descriptor that generates the maximum dot product is obtained from the second global splicing descriptors corresponding to different historical moments; and multiple historical images corresponding to the second global splicing descriptor that generates the maximum dot product are used as multiple historical target images.

[0099] S7. Based on the multiple current images and the multiple historical target images, determine the scene pose data of the local scene in the target scene.

[0100] Specifically, the two-dimensional feature points in the current image collected by each camera are matched with the three-dimensional feature points in the historical target image collected by each camera to obtain the matching point pairs of each camera; based on the matching point pairs of each camera, the scene pose data of the local scene in the target scene is determined.

[0101] Among them, the two-dimensional feature points in the current image collected by each camera are matched with the three-dimensional feature points in the historical target images collected by each camera to obtain the matching point pairs of each camera, including: based on the first local descriptor of the two-dimensional feature points in the current image collected by each camera and the second local descriptor of the two-dimensional feature points in the historical target images collected by each camera, the two-dimensional feature points in the current image collected by each camera are matched with the two-dimensional feature points in the historical target images collected by each camera to determine the two-dimensional feature point pairs; based on the two-dimensional feature point pairs and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target images, the matching point pairs are determined.

[0102] Among them, based on the matching point pairs of each camera, the scene pose data of the local scene in the target scene is determined, including: using a preset error equation to calculate the error of the first pose data corresponding to the two-dimensional feature points in multiple matching point pairs and the second pose data corresponding to the three-dimensional feature points to obtain a current error value; iteratively adjusting the current error value until the current error value meets the iteration stop condition, obtaining the target transformation pose between the two-dimensional feature points and the three-dimensional feature points in the multiple matching point pairs, and using the target transformation pose as the scene pose data of the local scene in the target scene.

[0103] Based on the same inventive concept as the above method embodiment, the present disclosure also provides a visual positioning device, referring to Figure 3 , is a schematic diagram of the structure of a visual positioning device provided in an embodiment of the present disclosure, wherein the visual positioning device 300 comprises:

[0104] A first determination module 310 is used to determine first global descriptors corresponding to the multiple current images respectively when the multi-eye camera collects multiple current images at the current moment, wherein the scenes corresponding to the multiple current images are local scenes in the target scene;

[0105] A stitching module 320 is used to stitch the first global descriptors respectively corresponding to the multiple current images to determine the first global stitching descriptor of the multi-eye camera at the current moment;

[0106] An acquisition module 330 is used to acquire a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped by mapping a plurality of historical images respectively collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-camera at different historical moments;

[0107] A second determination module 340 is used to determine, from the multiple historical images respectively collected by the multi-eye camera at the different historical moments, multiple historical target images matching the multiple current images based on the first global stitching descriptor and the second global stitching descriptors respectively corresponding to the different historical moments;

[0108] The positioning module 350 is used to determine the scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

[0109] In an optional implementation, the splicing module 320 includes:

[0110] a splicing unit, configured to splice first global descriptors respectively corresponding to the multiple current images according to a camera identifier of each camera, to obtain a first candidate splicing descriptor of the multi-camera at the current moment;

[0111] A normalization unit is used to perform normalization processing on the first candidate splicing descriptor to obtain the first global splicing descriptor.

[0112] In an optional implementation manner, the visual sparse map further includes second global descriptors corresponding to the plurality of historical images respectively; the device further includes:

[0113] A first determination module is configured to, for a plurality of historical images collected at each historical moment, splice second global descriptors respectively corresponding to the plurality of historical images collected at the plurality of historical moments according to a camera identifier of each camera, to obtain a second candidate spliced ​​descriptor of the multi-camera at each historical moment;

[0114] The second determination module is used to normalize the second candidate splicing descriptors corresponding to the multi-camera at the different historical moments to obtain the second global splicing descriptor.

[0115] In an optional implementation manner, the second determining module 340 includes:

[0116] A calculation unit, configured to calculate a dot product of the first global concatenation descriptor and the second global concatenation descriptors corresponding to the different historical moments;

[0117] A first acquisition unit is used to acquire a second global splicing descriptor that generates a maximum dot product from the second global splicing descriptors corresponding to the different historical moments;

[0118] The second acquisition unit is used to use the multiple historical images corresponding to the second global splicing descriptor that generates the maximum dot product as the multiple historical target images.

[0119] In an optional implementation, the positioning module 350 includes:

[0120] A matching point pair determination unit is used to match the two-dimensional feature points in the current image collected by each camera with the three-dimensional feature points in the historical target image collected by each camera to obtain the matching point pairs of each camera;

[0121] A positioning unit is used to determine the scene pose data of the local scene in the target scene based on the matching point pairs of each camera.

[0122] In an optional implementation manner, the matching point pair determination unit is specifically used to:

[0123] Based on the first local descriptor of the two-dimensional feature points in the current image collected by each camera and the second local descriptor of the two-dimensional feature points in the historical target images collected by each camera, the two-dimensional feature points in the current image collected by each camera are matched with the two-dimensional feature points in the historical target images collected by each camera to determine the two-dimensional feature point pairs;

[0124] The matching point pair is determined based on the two-dimensional feature point pair and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target image.

[0125] In an optional implementation manner, the positioning unit is specifically used for:

[0126] Using a preset error equation, an error calculation is performed on the first pose data corresponding to the two-dimensional feature points and the second pose data corresponding to the three-dimensional feature points in the plurality of matching point pairs to obtain a current error value;

[0127] The current error value is iteratively adjusted until the current error value satisfies an iteration stop condition, a target transformation pose between the two-dimensional feature points and the three-dimensional feature points in the multiple matching point pairs is obtained, and the target transformation pose is used as the scene pose data of the local scene in the target scene.

[0128] The disclosed embodiment provides a visual positioning device. When a multi-eye camera captures multiple current images at a current moment, a first global descriptor corresponding to each of the multiple current images is determined, wherein the scenes corresponding to the multiple current images are local scenes in a target scene; the first global descriptors corresponding to each of the multiple current images are spliced ​​to determine the first global splicing descriptor of the multi-eye camera at the current moment; a visual sparse map corresponding to the target scene is obtained, wherein the visual sparse map is pre-mapped to multiple historical images captured by the multi-eye camera at different historical moments, and the visual sparse map includes second global splicing descriptors corresponding to each of the multi-eye camera at different historical moments; based on the first global splicing descriptor and the second global splicing descriptors corresponding to different historical moments, multiple historical target images matching the multiple current images are determined from the multiple historical images captured by the multi-eye camera at different historical moments; based on the multiple current images and the multiple historical target images, scene pose data of the local scene in the target scene is determined. Through the above method, the first global stitching descriptor of the multi-eye camera at the current moment and the second global stitching descriptor corresponding to the multi-eye camera at different historical moments can be used to determine multiple historical target images and multiple current images with matching relationships for visual positioning. In this way, the visual positioning based on the multi-eye camera is realized quickly and at low cost, and the accuracy of the visual positioning is improved.

[0129] In addition to the above-mentioned method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device implements the visual positioning method of the embodiments of the present disclosure.

[0130] The embodiments of the present disclosure also provide a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the visual positioning method of the embodiments of the present disclosure is implemented.

[0131] In addition, the present disclosure also provides a visual positioning device, see Figure 4 As shown, the visual positioning device may include:

[0132] Processor 401, memory 402, input device 403 and output device 404. The number of processors 401 in the visual positioning device can be one or more. Figure 4 In some embodiments of the present disclosure, the processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means, wherein: Figure 4 The example of connecting through bus is taken in the following.

[0133] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing of the visual positioning device by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 403 may be used to receive input digital or character information, and to generate signal input related to the observation user settings and function control of the visual positioning device.

[0134] Specifically in this embodiment, the processor 401 will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the applications stored in the memory 402, so as to realize the various functions of the above-mentioned visual positioning device.

[0135] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0136] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual positioning method, characterized in that: The method comprises: When the multi-eye camera collects multiple current images at the current moment, determine first global descriptors corresponding to the multiple current images respectively, wherein the scenes corresponding to the multiple current images are local scenes in the target scene; Splicing the first global descriptors respectively corresponding to the multiple current images to determine the first global splicing descriptor of the multi-eye camera at the current moment; Obtaining a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped by mapping a plurality of historical images respectively collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-camera at different historical moments; Based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, determining a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments; Based on the multiple current images and the multiple historical target images, scene pose data of the local scene in the target scene is determined.

2. The method according to claim 1, characterized in that The step of stitching the first global descriptors respectively corresponding to the multiple current images to determine the first global stitching descriptor of the multi-eye camera at the current moment includes: According to the camera identifier of each eye camera, splicing the first global descriptors corresponding to the multiple current images respectively, to obtain a first candidate splicing descriptor of the multi-eye camera at the current moment; The first candidate splicing descriptor is normalized to obtain the first global splicing descriptor.

3. The method according to claim 1, characterized in that The visual sparse map further includes second global descriptors corresponding to the plurality of historical images respectively; the method further includes: For the multiple historical images collected at each historical moment, according to the camera identifier of each camera, the second global descriptors corresponding to the multiple historical images collected at each historical moment are spliced ​​to obtain the second candidate splicing descriptors of the multi-camera at each historical moment; Normalize the second candidate splicing descriptors corresponding to the multi-camera at the different historical moments to obtain the second global splicing descriptor.

4. The method according to claim 1, characterized in that The determining, based on the first global stitching descriptor and the second global stitching descriptors corresponding to the different historical moments, a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments, comprises: Calculating the dot product of the first global splicing descriptor and the second global splicing descriptors corresponding to the different historical moments; Obtaining a second global concatenated descriptor that generates a maximum dot product from the second global concatenated descriptors corresponding to the different historical moments; The multiple historical images corresponding to the second global splicing descriptor that generates the maximum dot product are used as the multiple historical target images.

5. The method according to claim 1, characterized in that The determining, based on the multiple current images and the multiple historical target images, scene pose data of the local scene in the target scene comprises: Matching the two-dimensional feature points in the current image collected by each camera with the three-dimensional feature points in the historical target image collected by each camera to obtain matching point pairs of each camera; Based on the matching point pairs of each camera, the scene pose data of the local scene in the target scene is determined.

6. The method according to claim 5, characterized in that The method of matching the two-dimensional feature points in the current image collected by each camera with the three-dimensional feature points in the historical target image collected by each camera to obtain the matching point pairs of each camera includes: Based on the first local descriptor of the two-dimensional feature points in the current image collected by each camera and the second local descriptor of the two-dimensional feature points in the historical target images collected by each camera, the two-dimensional feature points in the current image collected by each camera are matched with the two-dimensional feature points in the historical target images collected by each camera to determine the two-dimensional feature point pairs; The matching point pair is determined based on the two-dimensional feature point pair and the three-dimensional feature points corresponding to the two-dimensional feature points in the historical target image.

7. The method according to claim 5, characterized in that The determining, based on the matching point pairs of each camera, the scene pose data of the local scene in the target scene comprises: Using a preset error equation, an error calculation is performed on the first pose data corresponding to the two-dimensional feature points and the second pose data corresponding to the three-dimensional feature points in the plurality of matching point pairs to obtain a current error value; The current error value is iteratively adjusted until the current error value satisfies an iteration stop condition, a target transformation pose between the two-dimensional feature points and the three-dimensional feature points in the multiple matching point pairs is obtained, and the target transformation pose is used as the scene pose data of the local scene in the target scene.

8. A visual positioning device, characterized in that: The device comprises: A first determination module is used to determine first global descriptors corresponding to the multiple current images respectively when the multi-eye camera collects multiple current images at the current moment, wherein the scenes corresponding to the multiple current images are local scenes in the target scene; A stitching module, used to stitch the first global descriptors respectively corresponding to the multiple current images, and determine the first global stitching descriptor of the multi-eye camera at the current moment; An acquisition module is used to acquire a visual sparse map corresponding to the target scene, wherein the visual sparse map is pre-mapped by a plurality of historical images respectively collected by the multi-camera at different historical moments, and the visual sparse map includes second global splicing descriptors respectively corresponding to the multi-camera at different historical moments; A second determination module is used to determine, based on the first global splicing descriptor and the second global splicing descriptors corresponding to the different historical moments, a plurality of historical target images matching the plurality of current images from the plurality of historical images respectively acquired by the multi-eye camera at the different historical moments; A positioning module is used to determine the scene pose data of the local scene in the target scene based on the multiple current images and the multiple historical target images.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 7.

10. A device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images