Visual positioning method of intelligent equipment, intelligent equipment and storage medium

Abstract

The invention relates to the technical field of automatic driving, particularly provides a visual positioning method of intelligent equipment, the intelligent equipment and a storage medium, and aims to solve the technical problem of poor accuracy of an existing positioning method. In order to achieve the purpose, the intelligent equipment is provided with a plurality of image acquisition devices, and the visual positioning method of the intelligent equipment comprises the steps that two frames of images are acquired, and each frame of image is acquired based on the plurality of image acquisition devices; respectively acquiring bird's-eye view angle features corresponding to the two frames of images; determining the geographic position proximity of the two frames of images based on the bird's-eye view angle features corresponding to the two frames of images; and when the geographic position proximity is smaller than a preset proximity threshold, determining the relative pose of the intelligent device based on the bird's-eye view angle features corresponding to the two frames of images. Therefore, the accuracy of visual positioning is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and specifically provides a visual positioning method for an intelligent device, an intelligent device, and a storage medium. Background Art

[0002] Pose estimation is a basic capability in the field of autonomous driving. It is a key technology for achieving tasks such as global map construction, online perception, and sensor calibration.

[0003] The existing pose estimation methods have the following problems: they are usually based on single-camera images and it is difficult to effectively utilize multi-camera information from the vehicle; existing visual positioning methods usually handle topological positioning and pose estimation in completely independent ways, which not only ignores the commonality between the two links, but also increases the resource burden during positioning operation; existing pose estimation methods usually need to be combined with other sensor observations or prior information to obtain the relative pose between two frames, otherwise they can only give scale-free pose estimation results, which limits the accuracy of pose estimation. Summary of the invention

[0004] In order to overcome the above defects, the present application is proposed to solve or at least partially solve the technical problem of poor accuracy of existing positioning methods. The present application provides a visual positioning method for a smart device, a smart device and a storage medium.

[0005] In a first aspect, the present application provides a visual positioning method for an intelligent device, wherein the intelligent device is provided with a plurality of image acquisition devices, and the method comprises:

[0006] Acquire two frames of images, wherein each frame of the image is acquired based on the acquisition of a plurality of the image acquisition devices;

[0007] Respectively obtaining bird's-eye view features corresponding to the two frames of images;

[0008] Determining the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images;

[0009] When the geographic location proximity is less than a preset proximity threshold, the relative position and posture of the smart device is determined based on the bird's-eye view features corresponding to the two frames of images.

[0010] In one embodiment of the present application, determining the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images includes:

[0011] Determine the global descriptors corresponding to the two frames of images respectively based on the bird's-eye view features corresponding to the two frames of images;

[0012] Calculating the Euclidean distance between the global descriptors corresponding to the two frames of images;

[0013] The geographic location proximity of the two frames of image is determined based on the Euclidean distance.

[0014] In one embodiment of the present application, determining the global descriptors corresponding to the two frames of images based on the bird's-eye view features corresponding to the two frames of images respectively includes:

[0015] Using the decoding layer to process the bird's-eye view features corresponding to the two frames of images respectively, to obtain high-dimensional compact features corresponding to the two frames of images;

[0016] The high-dimensional compact features corresponding to the two frames of images are pooled respectively to obtain global descriptors corresponding to the two frames of images.

[0017] In one embodiment of the present application, determining the relative position and posture of the smart device based on the bird's-eye view features corresponding to the two frames of images includes:

[0018] Obtaining a preset candidate displacement;

[0019] Performing a displacement transformation on a bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement;

[0020] Determine local descriptors corresponding to the two frames of images based on bird's-eye view features corresponding to the image frame after displacement transformation and the image frame without displacement change in the two frames of images;

[0021] Performing local feature matching on local descriptors corresponding to the two frames of images;

[0022] The relative position and posture of the smart device is determined based on the local feature matching result.

[0023] In one embodiment of the present application, performing displacement transformation on the bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement includes:

[0024] Performing a translation transformation on a bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement;

[0025] A preset value is used to fill the featureless area in the bird's-eye view feature after the translation transformation.

[0026] In one embodiment of the present application, the determining of local descriptors corresponding to the two frames of images based on the bird's-eye view features corresponding to the image frame after displacement transformation and the image frame without displacement change in the two frames of images respectively includes:

[0027] Converting the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement change in the two frames of images from the smart device coordinate system to the polar coordinate system, wherein the bird's-eye view features converted to the polar coordinate system include polar angle and polar diameter;

[0028] A self-attention layer is used to enhance the internal correlation of the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement change in the two frames of images in the polar coordinate system;

[0029] A multi-layer perceptron is used to process the bird's-eye view features corresponding to the two frames of images to enhance attention to the polar angles of the bird's-eye view features corresponding to the two frames of images, eliminate dependence on the polar diameters of the bird's-eye view features corresponding to the two frames of images, and obtain local descriptors corresponding to the two frames of images.

[0030] In one embodiment of the present application, performing local feature matching on the local descriptors corresponding to the two frames of images includes:

[0031] The first local descriptor is continuously rotated multiple times in a preset direction in the polar coordinate system, each time by a preset angle, until it is rotated to a preset position;

[0032] The matching residual between the first local descriptor and the second local descriptor corresponding to each rotation is calculated, and the matching residuals corresponding to multiple yaw angles and multiple yaw angles are used as the local feature matching result, wherein the yaw angle is the rotation angle of the first local descriptor relative to the original position, the first local descriptor is any one of the local descriptors corresponding to the two frames of images, and the second local descriptor is the other one of the local descriptors corresponding to the two frames of images.

[0033] In one embodiment of the present application, determining the relative position and posture of the smart device based on the local feature matching result includes:

[0034] Determine an optimal matching residual based on the matching residuals respectively corresponding to the multiple yaw angles;

[0035] Determine whether the optimal matching residual is less than a preset residual threshold;

[0036] If yes, determining the relative position and posture of the smart device based on the optimal matching residual;

[0037] Otherwise, obtain a new candidate displacement, redetermine the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judge whether the re-determined optimal matching residual is less than a preset residual threshold. If so, redetermine the relative position and posture of the smart device based on the re-determined optimal matching residual; otherwise, loop through the steps of "obtaining a new candidate displacement, redetermining the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judging whether the re-determined optimal matching residual is less than a preset residual threshold" until the optimal matching residual is less than the preset residual threshold, or the number of loops is greater than the preset number threshold, and then determine the relative position and posture of the smart device based on the optimal matching residual.

[0038] In one embodiment of the present application, before performing local feature matching on the local descriptors corresponding to the two frames of images, the method further includes:

[0039] Using a self-attention layer to enhance the contextual relationship within the local descriptors corresponding to the two frames of images respectively;

[0040] A cross-attention layer is used to enhance the contextual relationship between the local descriptors corresponding to the two image frames.

[0041] In a second aspect, a smart device is provided, comprising:

[0042] at least one processor;

[0043] and, a memory communicatively coupled to the at least one processor;

[0044] Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, it is the aforementioned visual positioning method for the smart device.

[0045] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute any of the aforementioned visual positioning methods for smart devices.

[0046] Solution 1. A visual positioning method for an intelligent device, characterized in that the intelligent device is provided with a plurality of image acquisition devices, and the method comprises:

[0047] Acquire two frames of images, wherein each frame of the image is acquired based on the acquisition of a plurality of the image acquisition devices;

[0048] Respectively obtaining bird's-eye view features corresponding to the two frames of images;

[0049] Determining the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images;

[0050] When the geographic location proximity is less than a preset proximity threshold, the relative position and posture of the smart device is determined based on the bird's-eye view features corresponding to the two frames of images.

[0051] Solution 2. The visual positioning method for a smart device according to Solution 1 is characterized in that determining the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images comprises:

[0052] Determine the global descriptors corresponding to the two frames of images respectively based on the bird's-eye view features corresponding to the two frames of images;

[0053] Calculating the Euclidean distance between the global descriptors corresponding to the two frames of images;

[0054] The geographic location proximity of the two frames of image is determined based on the Euclidean distance.

[0055] Solution 3. The visual positioning method of the smart device according to Solution 2 is characterized in that the global descriptors corresponding to the two frames of images are determined based on the bird's-eye view features corresponding to the two frames of images, respectively, including:

[0056] Using the decoding layer to process the bird's-eye view features corresponding to the two frames of images respectively, to obtain high-dimensional compact features corresponding to the two frames of images;

[0057] The high-dimensional compact features corresponding to the two frames of images are pooled respectively to obtain global descriptors corresponding to the two frames of images.

[0058] Solution 4. The visual positioning method of the smart device according to Solution 1 is characterized in that the step of determining the relative position and posture of the smart device based on the bird's-eye view features corresponding to the two frames of images comprises:

[0059] Obtaining a preset candidate displacement;

[0060] Performing a displacement transformation on a bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement;

[0061] Determine local descriptors corresponding to the two frames of images based on bird's-eye view features corresponding to the image frame after displacement transformation and the image frame without displacement change in the two frames of images;

[0062] Performing local feature matching on local descriptors corresponding to the two frames of images;

[0063] The relative position and posture of the smart device is determined based on the local feature matching result.

[0064] Solution 5. The visual positioning method of the smart device according to Solution 4 is characterized in that the step of performing displacement transformation on the bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement comprises:

[0065] Performing a translation transformation on a bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement;

[0066] A preset value is used to fill the featureless area in the bird's-eye view feature after the translation transformation.

[0067] Solution 6. The visual positioning method of the smart device according to Solution 4 is characterized in that the local descriptors corresponding to the two frames of images are determined based on the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement change in the two frames of images, respectively, including:

[0068] Converting the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement change in the two frames of images from the smart device coordinate system to the polar coordinate system, wherein the bird's-eye view features converted to the polar coordinate system include a polar angle and a polar diameter;

[0069] A self-attention layer is used to enhance the internal correlation of the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement change in the two frames of images in the polar coordinate system;

[0070] A multi-layer perceptron is used to process the bird's-eye view features corresponding to the two frames of images to enhance attention to the polar angles of the bird's-eye view features corresponding to the two frames of images, eliminate dependence on the polar diameters of the bird's-eye view features corresponding to the two frames of images, and obtain local descriptors corresponding to the two frames of images.

[0071] Solution 7. The visual positioning method for a smart device according to Solution 4 is characterized in that the local feature matching of the local descriptors corresponding to the two frames of images comprises:

[0072] The first local descriptor is continuously rotated multiple times in a preset direction in the polar coordinate system, each time by a preset angle, until it is rotated to a preset position;

[0073] The matching residual between the first local descriptor and the second local descriptor corresponding to each rotation is calculated, and the matching residuals corresponding to multiple yaw angles and multiple yaw angles are used as the local feature matching result, wherein the yaw angle is the rotation angle of the first local descriptor relative to the original position, the first local descriptor is any one of the local descriptors corresponding to the two frames of images, and the second local descriptor is the other one of the local descriptors corresponding to the two frames of images.

[0074] Solution 8. The visual positioning method for a smart device according to Solution 7 is characterized in that determining the relative position and posture of the smart device based on the local feature matching result comprises:

[0075] Determine an optimal matching residual based on the matching residuals respectively corresponding to the multiple yaw angles;

[0076] Determine whether the optimal matching residual is less than a preset residual threshold;

[0077] If yes, determining the relative position and posture of the smart device based on the optimal matching residual;

[0078] Otherwise, obtain a new candidate displacement, redetermine the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judge whether the re-determined optimal matching residual is less than a preset residual threshold. If so, redetermine the relative position and posture of the smart device based on the re-determined optimal matching residual; otherwise, loop through the steps of "obtaining a new candidate displacement, redetermining the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judging whether the re-determined optimal matching residual is less than a preset residual threshold" until the optimal matching residual is less than the preset residual threshold, or the number of loops is greater than the preset number threshold, and then determine the relative position and posture of the smart device based on the optimal matching residual.

[0079] Solution 9. The visual positioning method of the smart device according to Solution 4 is characterized in that before performing local feature matching on the local descriptors corresponding to the two frames of images, it also includes:

[0080] Using a self-attention layer to enhance the contextual relationship within the local descriptors corresponding to the two frames of images respectively;

[0081] A cross-attention layer is used to enhance the contextual relationship between the local descriptors corresponding to the two image frames.

[0082] Solution 10. A smart device, comprising:

[0083] at least one processor;

[0084] and, a memory communicatively coupled to the at least one processor;

[0085] The memory stores a computer program, and when the computer program is executed by the at least one processor, the visual positioning method for the smart device described in any one of Schemes 1 to 9 is implemented.

[0086] Solution 11. A computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the visual positioning method for an intelligent device described in any one of Solutions 1 to 9.

[0087] The above one or more technical solutions of this application have at least one or more of the following Beneficial effects:

[0088] The visual positioning method of the smart device provided by the present application specifically includes: acquiring two frames of images, wherein each frame of the image is based on the bird's-eye view features corresponding to the two frames of images acquired by multiple image acquisition devices; determining the geographical proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images; when the geographical proximity is less than a preset proximity threshold, determining the relative posture of the smart device based on the bird's-eye view features corresponding to the two frames of images. By making full use of the multi-perspective information of images acquired by multiple cameras, the dependence on other sensors or prior information is reduced, and an accurate posture estimation result is obtained, thereby improving the accuracy of posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The disclosure of the present application will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. In addition, similar numbers in the drawings are used to represent similar components, among which:

[0090] Figure 1 This is a schematic diagram of the main process of a visual positioning method for a smart device in one embodiment of the present application;

[0091] Figure 2 is a schematic diagram of extracting bird's-eye view features in one embodiment of the present application;

[0092] Figure 3 is a schematic diagram of obtaining a global descriptor in one embodiment of the present application;

[0093] Figure 4 is a schematic diagram of a bird's-eye view feature in a vehicle body coordinate system in one embodiment of the present application;

[0094] Figure 5 is a bird's-eye view feature obtained by transforming the bird's-eye view feature using the candidate displacement in one embodiment of the present application;

[0095] Figure 6 is a schematic diagram of obtaining a local descriptor in one embodiment of the present application;

[0096] Figure 7 This is a schematic diagram of a process for obtaining a local descriptor in one embodiment of the present application;

[0097] Figure 8 is a schematic diagram of performing local feature matching in one embodiment of the present application;

[0098] Fig. 9 is a schematic diagram of a process for determining the relative position and posture of a smart device in one embodiment of the present application;

[0099] Fig.10 is a schematic diagram of the structure of a visual positioning system in one embodiment of the present application;

[0100] Fig.11 It is a schematic diagram of the main structure of a visual positioning device in one embodiment of the present application;

[0101] Fig.12 It is a schematic diagram of the structure of a smart device in one embodiment of the present application. DETAILED DESCRIPTION

[0102] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0103] In the description of the present application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B or A and B. The singular terms "one" and "the" may also include plural forms.

[0104] The accuracy of the pose estimation corresponding to the conventional existing pose estimation method is poor. To this end, the present application proposes a visual positioning method for an intelligent device, an intelligent device and a storage medium.

[0105] The smart devices described in this application may include driving devices, smart vehicles, robots, aircraft and other devices.

[0106] In order to make the purpose, technical solution and advantages of the present application more clear, the visual positioning method of the smart device of the present application is described in detail below with reference to the accompanying drawings and embodiments, taking a vehicle as an example of a smart device. It should be understood by those skilled in the art that the specific embodiments described herein are only used to explain the present application, and are not used to limit the visual positioning method of the smart device described in the present application to use a vehicle as a smart device.

[0107] See attached Figure 1 , Figure 1 It is a flowchart of the main steps of a visual positioning method for a smart device according to an embodiment of the present application.

[0108] like Figure 1 As shown, the visual positioning method of the smart device in the embodiment of the present application mainly includes the following steps S10 to S40.

[0109] Step S10: Acquire two frames of images, wherein each frame of image is acquired based on multiple image acquisition devices.

[0110] The image acquisition device refers to a hardware device that is set on the smart device and is used to capture images. For example, a camera can be used as an example of the image acquisition device.

[0111] The two frames of images refer to two frames of images at different moments, and each frame of images includes multiple images from multiple perspectives acquired by multiple image acquisition devices.

[0112] Step S20: respectively obtaining bird's-eye view features corresponding to the two frames of images.

[0113] The bird's-eye view feature refers to the visual characteristics presented when observing the current scene or object vertically downward from a high place or above.

[0114] Step S30: determining the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images.

[0115] Geographic proximity refers to how close two images are geographically. If two images show almost the same or very close places, their geographic proximity is high.

[0116] Step S40: When the geographic location proximity is less than a preset proximity threshold, the relative position and posture of the smart device is determined based on the bird's-eye view features corresponding to the two frames of images.

[0117] The preset proximity threshold is a pre-set standard value, which can be adaptively adjusted according to the actual scenario, and is not specifically limited to this. For example, 1m, 2m, 3m, etc. can be used as examples of the preset proximity threshold.

[0118] Based on the above steps S10 to S40, firstly, two frames of images are acquired, wherein each frame of the image is acquired based on a plurality of image acquisition devices to obtain the bird's-eye view features corresponding to the two frames of images; the geographical proximity of the two frames of images is determined based on the bird's-eye view features corresponding to the two frames of images; when the geographical proximity is less than a preset proximity threshold, the relative posture of the smart device is determined based on the bird's-eye view features corresponding to the two frames of images. By making full use of the multi-perspective information of images acquired by multiple cameras, the dependence on other sensors or prior information is reduced, and an accurate posture estimation result is obtained, thereby improving the accuracy of posture estimation.

[0119] The above steps S20 to S40 are further explained below.

[0120] In the above step S20, for each frame of multi-view image (N, H, W, 3), N is the number of images, H and W are the height and width of the image, and 3 represents the three primary color channels. For example, Figure 2 It can be used as a schematic diagram for extracting bird's-eye view features of each frame of multi-view images.

[0121] Specific as Figure 2 As shown, the ResNet-101 network is first used to extract dense features (N, H', W', C) from multi-view images, where H' and W' are the height and width after ResNet-101 processing, and C is the number of feature channels.

[0122] Then, an initial bird's-eye view feature (X, Y, D) is constructed, where X and Y are the height and width of the initial bird's-eye view feature, and D is the number of feature channels. The initial bird's-eye view feature can be a bird's-eye view feature corresponding to a historical image frame, or a preset image feature.

[0123] The self-attention layer is used to enhance the contextual relationship within the initial bird's-eye view feature to obtain the enhanced initial bird's-eye view feature. The enhanced initial bird's-eye view feature is normalized. The multi-view image features are fused with the initial bird's-eye view feature using a deformable cross-attention layer to enhance the contextual relationship between the multi-view image features and the initial bird's-eye view feature, and the fused features are normalized. The normalized features are nonlinearly transformed using a multi-layer perceptron to extract higher-level feature representations. Finally, the final features are normalized and the final bird's-eye view feature is output.

[0124] The above is a further description of step S20 , and the following is a further description of step S30 .

[0125] The above step S30 can be implemented through the following steps S301 to S303.

[0126] Step S301: determining global descriptors corresponding to the two frames of images based on the bird's-eye view features corresponding to the two frames of images.

[0127] A global descriptor is a feature vector used to describe the entire image region.

[0128] In a specific embodiment of the present application, global descriptors corresponding to the two frames of images are respectively determined based on the bird's-eye view features corresponding to the two frames of images, including: using a decoding layer to process the bird's-eye view features corresponding to the two frames of images respectively to obtain high-dimensional compact features corresponding to the two frames of images; and pooling the high-dimensional compact features corresponding to the two frames of images respectively to obtain global descriptors corresponding to the two frames of images.

[0129] The decoder may include at least one residual layer, each residual layer is typically composed of a series of convolutional layers followed by batch normalization and activation functions (such as ReLU), and the input may skip certain layers and be directly added to the output, wherein the number of residual layers included in the decoder may be adaptively adjusted according to the actual scenario, and there is no specific limitation on this.

[0130] Specifically, a decoder consisting of at least one residual layer is used to process the bird's-eye view features. The residual layer gradually reduces the spatial dimension of the features (for example, reducing the width and height of the image) through a series of convolution operations, activation functions, and jump connections, while increasing the number of channels of the features (that is, the feature dimension), thereby obtaining high-dimensional compact features corresponding to each frame of the image, thereby improving the expressiveness of the features.

[0131] Pooling is a downsampling operation used to reduce the amount of data and retain the most important information. Max pooling or average pooling can be an example of such pooling.

[0132] Specifically, the bird's-eye view features after being processed by the decoding layer may still contain a lot of detailed information. In order to obtain a more compact and representative global descriptor, the bird's-eye view features can be further pooled using a pooling layer. This helps reduce computational complexity and helps capture the overall characteristics of the entire image.

[0133] For example, Figure 3As shown, taking the decoder including a residual layer as an example, before using the residual layer to process the bird's-eye view feature, the bird's-eye view feature can be converted from the smart device coordinate system to the polar coordinate system, which can simplify the calculation and improve the data processing efficiency. Then, in the polar coordinate system, the residual layer gradually reduces the spatial dimension of the bird's-eye view feature through a series of convolution operations, activation functions and jump connections, and increases the feature dimension of the bird's-eye view feature, thereby obtaining the high-dimensional compact features corresponding to each frame of the image. Finally, the pooling layer is used to pool the high-dimensional compact features corresponding to each frame of the image to obtain the global descriptor corresponding to each frame of the image.

[0134] Step S302: Calculate the Euclidean distance between the global descriptors corresponding to the two frames of images.

[0135] The Euclidean distance is used to measure the similarity between two vectors (such as global descriptors). Specifically, if two global descriptors are very similar, the Euclidean distance between them is small; conversely, if the two are very different, the Euclidean distance will be large.

[0136] By calculating the Euclidean distance between the global descriptors of two frames, the similarity between the geographical areas shown in the two images can be quantified.

[0137] Step S303: determining the geographic location proximity of two image frames based on the Euclidean distance.

[0138] Specifically, a standard threshold can be pre-set, and the standard threshold can be adaptively adjusted according to the actual usage scenario. When the calculated Euclidean distance is less than the standard threshold, it is considered that the geographical location proximity of the two frames of images is high. Conversely, when the Euclidean distance exceeds the standard threshold, it means that the geographical location proximity of the two images is low.

[0139] By using the global descriptors corresponding to the two frames and quantifying the differences using the Euclidean distance, it is possible to accurately measure whether the two locations are close, which helps to improve the accuracy of visual positioning.

[0140] The above is a further description of step S30 , and the following is a further description of step S40 .

[0141] The above step S40 can be implemented through the following steps S401 to S405.

[0142] Step S401: obtaining a preset candidate displacement.

[0143] The candidate displacement is a candidate displacement randomly obtained from the candidate displacement set.

[0144] The candidate displacement set includes multiple candidate displacements that match the current scene. The candidate displacement set may be pre-constructed, and may be adjusted according to specific time or environmental changes, such as adding new candidate displacements or deleting no longer applicable candidate displacements.

[0145] Step S402: performing displacement transformation on the bird's-eye view feature corresponding to any one of the two image frames based on the candidate displacement.

[0146] In a specific embodiment of the present application, a displacement transformation is performed on the bird's-eye view features corresponding to any one of the two frames of images based on the candidate displacement, including: performing a translation transformation on the bird's-eye view features corresponding to any one of the two frames of images based on the candidate displacement; and filling the featureless area in the bird's-eye view features after the translation transformation with a preset value.

[0147] Specifically, the bird's-eye view feature corresponding to any one of the two frames of images is selected for translation transformation. The specific one to be selected can be preset or can be selected according to actual needs, and there is no specific limitation on this.

[0148] The translation transformation refers to the operation of moving the bird's-eye view feature by a certain distance in both the horizontal and vertical directions. In this embodiment, the candidate displacement determines the direction and distance of the translation.

[0149] The scale of the bird's-eye view image should be clear, that is, how much distance each pixel in the image represents in reality (such as how many meters each pixel corresponds to), where the scale depends on factors such as the resolution of the shooting device, focal length, and shooting height. For example, if each pixel represents 1 meter, then a real displacement of 1 meter in the image is equivalent to a distance of 1 pixel.

[0150] The translation transformation of the bird's-eye view feature will cause some of the features to move out of the boundary, and will also leave blank areas at the original position. These blank areas are featureless areas. In order to fill these featureless areas, preset values ​​are used for filling. The selection of preset values ​​can be determined according to the specific application scenario. For example, 0, 1, etc. can be used as examples of the preset values.

[0151] Padding ensures that the image size remains the same and avoids white space that can cause problems in subsequent processing steps. In addition, proper padding can help maintain the overall coherence and consistency of the image.

[0152] For example, taking a vehicle as an example of a smart device, Figure 4 It can be used as a schematic diagram of the bird's-eye view feature in the vehicle body coordinate system (where the X axis represents the vehicle's forward direction and the Y axis represents the direction perpendicular to the vehicle's forward direction), and the bird's-eye view feature is displaced. Since the candidate displacement is positive, Figure 4The bird's-eye view feature in is translated along the X-axis and Y-axis, such as Figure 5 As shown, the featureless area after translation can be filled with 1.

[0153] Step S403: determining local descriptors corresponding to the two image frames based on the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement transformation in the two image frames.

[0154] In a specific embodiment of the present application, local descriptors corresponding to the two frames of images are determined based on the bird's-eye view features corresponding to the image frames after the displacement transformation and the image frames without the displacement change in the two frames of images, including: converting the bird's-eye view features corresponding to the image frames after the displacement transformation and the image frames without the displacement change in the two frames of images from the smart device coordinate system to the polar coordinate system, wherein the bird's-eye view features converted to the polar coordinate system include polar angles and polar diameters; using a self-attention layer to enhance the internal correlation of the bird's-eye view features corresponding to the image frames after the displacement transformation and the image frames without the displacement change in the two frames of images in the polar coordinate system; using a multi-layer perceptron to process the bird's-eye view features corresponding to the two frames of images to enhance attention to the polar angles of the bird's-eye view features corresponding to the two frames of images, eliminate dependence on the polar diameters of the bird's-eye view features corresponding to the two frames of images, and obtain local descriptors corresponding to the two frames of images.

[0155] Local descriptors are used to represent the features of a specific region (i.e., local region) in an image. These features are usually extracted from key points, interest points, or salient regions in an image.

[0156] To facilitate subsequent processing, the bird's-eye view feature can be converted to a polar coordinate system through coordinate system transformation. In the polar coordinate system, the position of each pixel is determined by two parameters, namely the polar diameter (the distance to the origin) and the polar angle (the angle with a reference direction). Converting the bird's-eye view feature to the polar coordinate system helps to simplify the relative position relationship between features and improve the efficiency of subsequent feature matching.

[0157] Self-attention is a deep learning technique that allows the model to consider information about all other features when processing each feature.

[0158] After transforming the bird’s-eye view features into polar coordinates, a self-attention layer is used to enhance the internal correlation of the bird’s-eye view features.

[0159] The multi-layer perceptron is a feed-forward neural network that can learn complex mappings from input features to output features. By training the multi-layer perceptron, we can increase the focus on the polar angle of the bird's-eye view features corresponding to the two frames of images and eliminate the dependence on the polar diameter of the bird's-eye view features corresponding to the two frames of images.

[0160] After processing the bird's-eye view features corresponding to the image frames after displacement transformation and the image frames without displacement changes in the two frames of images in sequence through the self-attention layer and the multi-layer perceptron, we finally obtain the local descriptors corresponding to the two frames of images.

[0161] In a specific embodiment of the present application, before performing local feature matching on the local descriptors corresponding to the two frames of images, it also includes: using a self-attention layer to enhance the contextual relationship within the local descriptors corresponding to the two frames of images respectively; and using a cross-attention layer to enhance the contextual relationship between the local descriptors corresponding to the two frames of images.

[0162] The self-attention layer enables the model to focus on the relationship between different parts of the input features. It allows the model to dynamically adjust the degree of attention to each element based on the context. The self-attention layer can enhance the internal correlation between local descriptors in each frame of the image to focus on features that are critical to describing the scene.

[0163] Cross-attention is another attention mechanism that allows information exchange between two local descriptors. Through the cross-attention layer, the contextual relationship between the local descriptors of the two frames can be enhanced to ensure that they are represented consistently during the matching process. In this way, the most similar feature points can be accurately located while considering the global context, which helps to improve the matching accuracy between the two local descriptors.

[0164] For example, Figure 6 and Figure 7 It can be used as a schematic diagram to obtain local descriptors. Figure 6 First, the bird's-eye view features after displacement transformation are transformed into polar coordinates to obtain the bird's-eye view features under polar coordinates, and then the bird's-eye view features are processed using the self-attention layer and the multi-layer perceptron to obtain the local descriptor.

[0165] Step S404: performing local feature matching on the local descriptors corresponding to the two frames of images.

[0166] In a specific embodiment of the present application, local feature matching is performed on the local descriptors corresponding to the two frames of images, including: continuously rotating the first local descriptor in a polar coordinate system in a preset direction for multiple times, each time rotating by a preset angle, until it is rotated to a preset position; calculating the matching residual between the first local descriptor and the second local descriptor corresponding to each rotation, and taking a plurality of yaw angles and the matching residuals corresponding to the plurality of yaw angles as the local feature matching result, wherein the yaw angle is the rotation angle of the first local descriptor relative to the original position, the first local descriptor is any one of the local descriptors corresponding to the two frames of images, and the second local descriptor is another one of the local descriptors corresponding to the two frames of images.

[0167] The preset angle may be a preset rotation angle value, which may be adaptively adjusted according to the actual usage scenario. For example, 30°, 60°, and 90° may be used as examples of the preset angle.

[0168] The preset position may be a preset position at the end of the rotation. For example, a rotation of 360° may be used as an example of the preset position.

[0169] Specifically, any one of the local descriptors corresponding to the two frames of images is selected as the first local descriptor, and the other is selected as the second local descriptor. The first local descriptor is continuously rotated by a preset angle in a preset direction (for example, clockwise) in a polar coordinate system, and rotated multiple times until it is rotated to a preset position. The rotation angle of the first local descriptor after each rotation relative to the original position is used as the yaw angle. For each rotation, the matching residual between the rotated first local descriptor and the second local descriptor is calculated. The matching residual can be represented by any one of Euclidean distance, Manhattan distance, cosine similarity, etc. For example, Figure 8 This can be used as an example of matching the first local descriptor with the second local descriptor.

[0170] After local feature matching, multiple yaw angles and multiple matching residuals are obtained.

[0171] Step S405: Determine the relative position and posture of the smart device based on the local feature matching result.

[0172] Specifically, step S405 can be implemented through the following steps S4051 to S4054.

[0173] Step S4051: determining the optimal matching residual based on the matching residuals corresponding to the multiple yaw angles.

[0174] Specifically, firstly, the results obtained by the above-mentioned local matching (including candidate displacements, multiple yaw angles corresponding to the candidate displacements, and multiple matching residuals) are stored in a result pool, and then the minimum matching residual is searched in the result pool as the optimal matching residual.

[0175] Step S4052: Determine whether the best matching residual is less than a preset residual threshold.

[0176] The preset residual threshold is a preset value determined in advance through experiments, and can be adaptively adjusted according to actual scenarios.

[0177] Specifically, first determine whether the best matching residual is less than the preset residual threshold; when the best matching residual is less than the preset residual threshold, execute the following step S4053; when the best matching residual is not less than the preset residual threshold, execute the following step S4054.

[0178] Step S4053: Determine the relative position and posture of the smart device based on the optimal matching residual.

[0179] Specifically, the yaw angles of the two frames of images corresponding to the optimal matching residual are obtained, wherein the yaw angles of the two frames of images are the relative posture of the smart device.

[0180] Step S4054: obtaining a new candidate displacement, re-determining the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judging whether the re-determined optimal matching residual is less than a preset residual threshold; if so, re-determining the relative position and posture of the smart device based on the re-determined optimal matching residual; otherwise, looping the step of "obtaining a new candidate displacement, re-determining the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judging whether the re-determined optimal matching residual is less than a preset residual threshold" until the optimal matching residual is less than the preset residual threshold, or the number of loops is greater than the preset number threshold, and then determining the relative position and posture of the smart device based on the optimal matching residual.

[0181] The new candidate displacement is another candidate displacement randomly obtained from the candidate displacement set.

[0182] The number of cycles refers to the number of times the logic cycle judgment is repeatedly executed. In this embodiment, the number of cycles can be determined by the number of times the candidate displacement is obtained.

[0183] The preset residual threshold may be a preset value determined in advance through experiments, and may be adaptively adjusted according to actual scenarios.

[0184] Specifically, when the optimal matching residual is greater than the preset residual threshold, a new candidate displacement is randomly obtained from the displacement set, and the above steps S402 to S404 are executed according to the new candidate displacement to obtain the optimal matching residual. And it is judged whether the re-determined optimal matching residual is less than the preset residual threshold. If so, the yaw angle of the two frames of images corresponding to the re-determined optimal matching residual is used as the relative posture of the smart device. Otherwise, the above-mentioned "obtaining a new candidate displacement, re-determining the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judging whether the re-determined optimal matching residual is less than the preset residual threshold" step is executed repeatedly until the optimal matching residual is less than the preset residual threshold, or the number of cycles is greater than the preset number threshold, and the yaw angle of the two frames of images corresponding to the re-determined optimal matching residual is used as the relative posture of the smart device.

[0185] By obtaining candidate displacements and performing displacement transformation on the bird's-eye view features, and then performing feature matching based on local descriptors, the relative position of the smart device can be determined more accurately. The spatial relationship between images is taken into account, the matching error is reduced, and the positioning accuracy is improved.

[0186] For example, Fig. 9 It can be used as a complete flow chart for determining the relative position and posture of the smart device. Specifically, the relative position and posture of the smart device can be determined through the following steps S1-S8.

[0187] Step S1: Obtain candidate displacements from a candidate displacement set using an enumerator.

[0188] Step S2: Use the obtained candidate displacement to perform a translation transformation on any one of the bird's-eye view features corresponding to the two frames of images, for example Fig. 9 The candidate displacement is used to perform translation transformation on the bird's-eye view feature A. The specific transformation process can be referred to the aforementioned step S402, which will not be described here.

[0189] Step S3: Obtain local descriptors of the bird's-eye view features corresponding to the two frames of images, wherein the local descriptor corresponding to the bird's-eye view feature A is local descriptor A, and the local descriptor corresponding to the bird's-eye view feature B is local descriptor B. The specific process of determining the local descriptor can refer to the aforementioned step S403, which will not be repeated here.

[0190] Step S4: Use the self-attention layer and the cross-attention layer to process the local descriptors corresponding to the two frames of images respectively to improve the contextual relationship within the local descriptors and between the local descriptors.

[0191] Step S5: Perform local feature matching on the two local descriptors to obtain a local feature matching result, wherein the local feature matching result includes a matching residual and a yaw angle corresponding to the matching residual. The specific matching process can be referred to the aforementioned step S404, which will not be described here.

[0192] Step S6: adding the matching residual, the yaw angle corresponding to the matching residual, and the candidate displacement to a result pool, and searching the result pool for the minimum matching residual as the optimal matching residual.

[0193] Step S7: Determine whether the optimal matching residual is less than a preset residual threshold, or whether the number of cycles is greater than a preset number threshold. If so, execute the following step S8; otherwise, return and repeat the above steps S1-S7.

[0194] Step S8: Determine the relative posture of the smart device based on the optimal matching residual, specifically, obtain the yaw angle of the two frames of images corresponding to the optimal matching residual, and output the yaw angle of the two frames of images as the relative posture of the smart device.

[0195] In addition, if Fig.10 As shown, the present application also provides a visual positioning system, which includes a bird's-eye view feature extraction module, a global descriptor decoding module, a topological positioning module, a local descriptor decoding module and a metric positioning module. Among them, the bird's-eye view extraction module is used to extract the bird's-eye view features of two frames of multi-camera images respectively. The global descriptor decoding module is used to extract the global descriptors of the two frames of images respectively according to the bird's-eye view features. The topological positioning module is used to calculate the geographical location proximity of the two frames of images based on the global descriptors of the two frames of images. When the geographical location proximity of the two frames of images is less than a preset proximity threshold, the local descriptors of the two frames of images are obtained through the local descriptor decoding module. The metric positioning module is used to perform local feature matching based on two local descriptors, and determine the relative posture of the smart device based on the local feature matching results.

[0196] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present application.

[0197] Furthermore, the present application also provides a visual positioning device.

[0198] See attached Fig.11 , Fig.11 FIG. 1 is a main structural block diagram of a visual positioning device according to an embodiment of the present application. Fig.11 As shown, the visual positioning device in the embodiment of the present application mainly includes a first acquisition module 11, a second acquisition module 12, a first determination module 13 and a second determination module 14. In some embodiments, one or more of the first acquisition module 11, the second acquisition module 12, the first determination module 13 and the second determination module 14 can be combined into one module.

[0199] In some embodiments, the first acquisition module 11 may be configured to acquire two frames of images, wherein each frame of the image is acquired based on multiple image acquisition devices.

[0200] The second acquisition module 12 may be configured to respectively acquire bird's-eye view features corresponding to the two frames of images.

[0201] The first determination module 13 may be configured to determine the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images.

[0202] The second determination module 14 may be configured to determine the relative position and posture of the smart device based on the bird's-eye view features corresponding to the two frames of images when the geographic location proximity is less than a preset proximity threshold.

[0203] In one implementation, the description of the specific implementation functions can refer to steps S10-S40.

[0204] The visual positioning device is used to perform Figure 1 The visual positioning method embodiment shown in the figure has similar technical principles, technical problems solved and technical effects produced. Technical personnel in this technical field can clearly understand that for the convenience and conciseness of description, the specific working process and related instructions of the visual positioning device can refer to the contents described in the embodiment of the visual positioning method, which will not be repeated here.

[0205] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present application, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0206] It is understood by those skilled in the art that each module in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present application, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present application.

[0207] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0208] Furthermore, the present application also provides an intelligent device, which may include at least one processor; and a memory connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the visual positioning method described in any of the above embodiments is implemented. Fig.12 As shown, Fig.12 exemplarily shows the structure of a smart device, which includes a processor 100 and a memory 200.

[0209] In some embodiments of the present application, the smart device further includes at least one sensor, which is used to sense information. The sensor is communicatively connected to any type of processor mentioned in the present application. Optionally, the smart device further includes an autonomous driving system, which is used to guide the smart device to drive itself or assist in driving. The processor communicates with the sensor and / or the autonomous driving system to complete the visual positioning method described in any of the above embodiments.

[0210] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store a program for executing the visual positioning method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned visual positioning method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a memory device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0211] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0212] The user personal information processed by this application will vary depending on the specific product / service scenario, and will be based on the specific scenario in which the user uses the product / service. It may involve the user's account information, device information, driving information, vehicle information or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0213] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0214] So far, the technical solutions of the present application have been described in conjunction with the specific implementations shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific implementations. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A visual positioning method for an intelligent device, characterized in that: The smart device is provided with a plurality of image acquisition devices, and the method comprises: Acquire two frames of images, wherein each frame of the image is acquired based on the acquisition of a plurality of the image acquisition devices; Respectively obtaining bird's-eye view features corresponding to the two frames of images; Determining the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images; When the geographic location proximity is less than a preset proximity threshold, the relative position and posture of the smart device is determined based on the bird's-eye view features corresponding to the two frames of images.

2. The visual positioning method of the smart device according to claim 1, characterized in that: The determining the geographic location proximity of the two frames of images based on the bird's-eye view features corresponding to the two frames of images includes: Determine the global descriptors corresponding to the two frames of images respectively based on the bird's-eye view features corresponding to the two frames of images; Calculating the Euclidean distance between the global descriptors corresponding to the two frames of images; The geographic location proximity of the two frames of image is determined based on the Euclidean distance.

3. The visual positioning method of the smart device according to claim 2, characterized in that: The determining the global descriptors corresponding to the two frames of images respectively based on the bird's-eye view features corresponding to the two frames of images includes: Using the decoding layer to process the bird's-eye view features corresponding to the two frames of images respectively, to obtain high-dimensional compact features corresponding to the two frames of images; The high-dimensional compact features corresponding to the two frames of images are pooled respectively to obtain global descriptors corresponding to the two frames of images.

4. The visual positioning method of the smart device according to claim 1, characterized in that: The determining the relative position and posture of the smart device based on the bird's-eye view features corresponding to the two frames of images includes: Obtaining a preset candidate displacement; Performing a displacement transformation on a bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement; Determine local descriptors corresponding to the two frames of images based on bird's-eye view features corresponding to the image frame after displacement transformation and the image frame without displacement change in the two frames of images; Performing local feature matching on local descriptors corresponding to the two frames of images; The relative position and posture of the smart device is determined based on the local feature matching result.

5. The visual positioning method of the smart device according to claim 4, characterized in that: The performing displacement transformation on the bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement includes: Performing a translation transformation on a bird's-eye view feature corresponding to any one of the two frames of images based on the candidate displacement; A preset value is used to fill the featureless area in the bird's-eye view feature after the translation transformation.

6. The visual positioning method of the smart device according to claim 4, characterized in that: The determining of local descriptors corresponding to the two frames of images based on the bird's-eye view features corresponding to the image frame after displacement transformation and the image frame without displacement change in the two frames of images respectively includes: Converting the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement change in the two frames of images from the smart device coordinate system to the polar coordinate system, wherein the bird's-eye view features converted to the polar coordinate system include a polar angle and a polar diameter; A self-attention layer is used to enhance the internal correlation of the bird's-eye view features corresponding to the image frame after the displacement transformation and the image frame without the displacement change in the two frames of images in the polar coordinate system; A multi-layer perceptron is used to process the bird's-eye view features corresponding to the two frames of images to enhance attention to the polar angles of the bird's-eye view features corresponding to the two frames of images, eliminate dependence on the polar diameters of the bird's-eye view features corresponding to the two frames of images, and obtain local descriptors corresponding to the two frames of images.

7. The visual positioning method of the smart device according to claim 4, characterized in that: The performing local feature matching on the local descriptors corresponding to the two frames of images includes: The first local descriptor is continuously rotated multiple times in a preset direction in the polar coordinate system, each time by a preset angle, until it is rotated to a preset position; The matching residual between the first local descriptor and the second local descriptor corresponding to each rotation is calculated, and the matching residuals corresponding to multiple yaw angles and multiple yaw angles are used as the local feature matching result, wherein the yaw angle is the rotation angle of the first local descriptor relative to the original position, the first local descriptor is any one of the local descriptors corresponding to the two frames of images, and the second local descriptor is the other one of the local descriptors corresponding to the two frames of images.

8. The visual positioning method of the smart device according to claim 7, characterized in that: The determining the relative position and posture of the smart device based on the local feature matching result includes: Determine an optimal matching residual based on the matching residuals respectively corresponding to the multiple yaw angles; Determine whether the optimal matching residual is less than a preset residual threshold; If yes, determining the relative position and posture of the smart device based on the optimal matching residual; Otherwise, obtain a new candidate displacement, re-determine the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judge whether the re-determined optimal matching residual is less than a preset residual threshold; if so, re-determine the relative position and posture of the smart device based on the re-determined optimal matching residual; otherwise, loop through the steps of "obtaining a new candidate displacement, re-determining the optimal matching residual based on the new candidate displacement and the bird's-eye view features corresponding to the two frames of images, and judging whether the re-determined optimal matching residual is less than a preset residual threshold" until the optimal matching residual is less than the preset residual threshold, or the number of loops is greater than the preset number threshold, and then determine the relative position and posture of the smart device based on the optimal matching residual.

9. The visual positioning method of the smart device according to claim 4, characterized in that: Before performing local feature matching on the local descriptors corresponding to the two frames of images, the method further includes: Using a self-attention layer to enhance the contextual relationship within the local descriptors corresponding to the two frames of images respectively; A cross-attention layer is used to enhance the contextual relationship between the local descriptors corresponding to the two image frames.

10. A smart device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores a computer program, and when the computer program is executed by the at least one processor, the visual positioning method for the smart device according to any one of claims 1 to 9 is implemented.

Images