Visual position recognition method for cross-modal retrieval, storage medium and electronic device

Abstract

The application provides a visual position recognition method for cross-modal retrieval, a storage medium and an electronic device, and the method comprises the following steps: constructing a 3D point cloud map database in an offline manner at one time, converting dense point cloud representation in the database into point cloud in a lightweight structure form, and fusing context information for the lightweight point cloud by using an attention mechanism to obtain global features of the 3D point cloud map database; obtaining a to-be-queried image in real time in an online manner, extracting global features of the to-be-queried 2D image by using a lightweight image backbone network; calculating the Euclidean distance of the global features of the 3D point cloud map database and the global features of the 2D image, and establishing the corresponding relationship therebetween. Then, the features of the 2D to-be-queried image can be used to real-time query the features of the 3D point cloud database represented in a lightweight structure, so as to obtain a point cloud map in the same geographical position as the to-be-queried image, and the feature information of the query image can be quickly extracted.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a visual position recognition method, storage medium, and electronic device for cross-modal retrieval. Background Art

[0002] In recent years, the development of deep convolutional neural networks has greatly promoted the progress of related algorithms in the field of computer vision. As one of the research branches in the field of computer vision, Visual Place Recognition's main task is to locate the current visual position and identify where it is currently located in the map database. The visual position recognition task is an important component of the simultaneous localization and mapping system, which is used to solve the loop-closure detection problem. Usually, positioning tasks in weak GPS environments mainly rely on the simultaneous localization and mapping system, which locates the current position of the device through the map database built by the device, and improves the map database through continuous positioning, making outstanding contributions in applications such as unmanned driving, robotics, and augmented reality. Loop closure detection is used to determine whether the current position has been reached before in order to detect loops.

[0003] Typically, location information acquired by sensors on devices is divided into two main data types: 2D images and 3D point clouds. 2D images are captured directly by cameras, while 3D point clouds are generated by LiDAR scanning scenes. Images are highly susceptible to lighting, weather, and seasonality, which can lead to reduced recognition accuracy. LiDAR is expensive to install and difficult to maintain, making it costly in practical applications. Currently, the main visual location recognition methods used include those based on 2D images and those based on 3D point clouds.

[0004] This 2D image-based visual location recognition method generates a query image and a database image using camera capture. It then uses a classic image backbone network to extract features from the query and database images. This extracted feature is aggregated into global features using a Net Vector of Locally Aggregated Descriptors (NVAD) layer. A simple Euclidean distance is then used to retrieve database images that match the query image. Vehicle-mounted cameras are simple to install, low-cost, and easy to maintain.

[0005] The 3D point cloud-based visual location recognition method generates query and database point clouds through on-board LiDAR scanning. A classic point cloud backbone network is used to extract features from the query and database point clouds. The extracted features are aggregated into global features using a Net Vector of Locally Aggregated Descriptors (NVAD) layer. A fully connected layer is used to reduce the dimensionality of the global features and compress the feature vectors for efficient retrieval. Finally, Euclidean distance is used to retrieve a point cloud image from the database point cloud that matches the query point cloud. The on-board LiDAR scans the scene, unaffected by lighting, weather, or season. Even at night, it can fully capture the scene cloud image, ensuring high accuracy in location recognition.

[0006] However, visual location recognition methods based on 2D images are easily affected by lighting, resulting in very low accuracy at night. Furthermore, accuracy is also susceptible to seasonal variations. While visual location recognition methods based on 3D point clouds can effectively address the challenges posed by scene appearance changes, they require expensive LiDAR equipment, which is difficult to implement and maintain. Summary of the Invention

[0007] Therefore, it is necessary to provide a technical solution for visual position recognition for cross-modal retrieval to solve the problems of poor recognition accuracy and high maintenance cost of the visual position recognition methods involved in the existing technology.

[0008] To achieve the above objectives, in a first aspect, the present invention provides a visual location recognition method for cross-modal retrieval, the method comprising the following steps:

[0009] S1: When constructing a 3D point cloud map database in an offline manner, the dense point cloud is compressed into the first lightweight structure point cloud;

[0010] S2: inputting the first lightweight structure point cloud into a rotation learning network to generate a second lightweight structure point cloud;

[0011] S3: Inputting the second lightweight structure point cloud into an attention feature processing module, and using the attention feature processing module to fuse context information for the second lightweight structure point cloud using an attention mechanism to obtain a third lightweight structure point cloud with attention weights;

[0012] S4: aggregating the third lightweight structure point cloud through a pooling layer to obtain global features of the 3D point cloud map database;

[0013] S5: obtaining a query image online, extracting global features of the query image through a lightweight image backbone network, and inputting the global features of the query image into a pooling layer to aggregate them into 2D image global features;

[0014] S6: Calculate the Euclidean distance between the global features of the 2D image and the global features of the 3D point cloud map database, perform feature matching between the two, and establish a corresponding relationship between the 2D image and the 3D point cloud.

[0015] In some embodiments, the first lightweight structure point cloud comprises a point cloud in a normal distribution transform (NDT) format;

[0016] The compressing the dense point cloud into a first lightweight structure point cloud includes:

[0017] The dense point cloud is compressed into a point cloud in the normal distribution transform (NDT) form according to the following formula:

[0018]

[0019] Among them, μ represents the mathematical expectation, σ represents the variance, n represents the number of 3D points for normal distribution transformation, X k represents the kth 3D point transformed under the normal distribution.

[0020] In some embodiments, in step S2, the rotation learning network generates a second lightweight structure point cloud according to the following formula:

[0021]

[0022] Among them, T 3×3 Represents a 3×3 rotation matrix generated by the rotation learning network.

[0023] In some embodiments, the attention feature processing module includes a plurality of attention encoders connected in series, and a shared fully connected layer with skip connections is provided at the head and tail of the attention feature processing module;

[0024] Each of the attention encoders is configured to: calculate weights using a multi-head self-attention layer to obtain 3D points transformed by a normal distribution with attention weights, and input the obtained 3D points transformed by a normal distribution into a feedforward network, and then perform normalization processing.

[0025] In some embodiments, step S4 may be followed by step S41: inputting the global features of the 3D point cloud map database obtained in step S4 into a multilayer perceptron for processing to obtain 3D point cloud global features of consistent size.

[0026] In certain embodiments, the method comprises:

[0027] A loss function is formed based on the positive and negative samples and the randomly sampled samples in the 2D image and 3D point cloud map database, and the parameters of the visual position recognition network are updated by the loss function;

[0028] The updating of the visual position recognition network parameters includes:

[0029] performing parameter updating on the rotation learning network;

[0030] and / or updating parameters of the attention feature processing module;

[0031] and / or updating parameters of the lightweight image backbone network;

[0032] And / or update parameters of the pooling layer.

[0033] In some embodiments, the loss function is a quadruple loss function, and the formula of the quadruple loss function is as follows:

[0034]

[0035] Among them, ξ q represents the global features of the query image, ξ p represents the global features of the positive samples in the point cloud database, ξ n Represents the global features of negative samples in the point cloud database, represents the global features randomly sampled in the query image and point cloud database, and is related to ξ q ,ξ p and ξ n are all different, d() represents the Euclidean distance, H() represents the hinge loss function, and α and β represent margin constants.

[0036] In certain embodiments, the method comprises:

[0037] Retrieving a point cloud map corresponding to the image to be queried from the 3D point cloud map database;

[0038] And the global features of the image to be queried and the global features of the positive and negative samples in the 3D point cloud map database are trained and learned to improve the accuracy of the 3D point cloud map database retrieved in real time based on the image to be queried; the image to be queried is a 2D image.

[0039] In a second aspect, the present invention further provides a storage medium storing a computer program, which, when executed, implements the method described in the first aspect of the present invention.

[0040] In a third aspect, the present application also provides a server, comprising:

[0041] The storage medium is the storage medium described in the second aspect of the present application.

[0042] The processor is electrically connected with the storage medium, and is used for executing the computer program stored in the storage medium to realize the method described in the first aspect of the present application.

[0043] Compared with the prior art, the present application has the following characteristics:

[0044] The present application provides a cross-modal retrieval visual position recognition method, a storage medium and an electronic device, the method comprising: constructing a 3D point cloud map database in an offline manner at one time, converting the dense point cloud representation in the database into a point cloud in a lightweight structure form, and utilizing an attention mechanism to fuse context information for the lightweight point cloud to obtain the global features of the map database; obtaining a to-be-queried image in real time in an online manner, extracting the global features of the to-be-queried image through a lightweight image backbone network, and establishing a corresponding relationship therebetween. Then, the features of the 2D to-be-queried image can be used to real-time query the 3D point cloud database features represented in a lightweight structure to obtain a point cloud map in the same geographical position as the to-be-queried image, so that the feature information of the query image can be quickly extracted, and the recognition accuracy and the retrieval speed of the model are improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the cross-modal retrieval visual position recognition method according to the first embodiment of the present application;

[0046] Figure 2 The flowchart of the cross-modal retrieval visual position recognition method according to the second embodiment of the present application;

[0047] Figure 3 The schematic diagram of the attention feature processing module according to an embodiment of the present application;

[0048] Figure 4 The schematic diagram of the overall learning flow of the cross-modal retrieval visual position recognition according to an embodiment of the present application;

[0049] Figure 5 The structural schematic diagram of the cross-modal retrieval visual position recognition device according to an embodiment of the present application;

[0050] Figure 6 The schematic diagram of the electronic device according to an embodiment of the present application.

[0051] REFERENCE NUMERALS:

[0052] 10, electronic device;

[0053] 101. Processor;

[0054] 102. Storage media;

[0055] 1. Conversion module;

[0056] 2. First sending module;

[0057] 3. Second sending module;

[0058] 4. The first generation module;

[0059] 5. Second generation module;

[0060] 6. Learning module;

[0061] 7. Update module. DETAILED DESCRIPTION

[0062] In order to explain in detail the possible application scenarios, technical principles, specific solutions that can be implemented, and the purpose and effects of this application, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this application and are therefore only examples and are not intended to limit the scope of protection of this application.

[0063] References to "embodiments" herein mean that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the word "embodiment" in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or relevance to other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the various technical features mentioned in the embodiments can be combined in any manner to form a corresponding implementable technical solution.

[0064] Unless otherwise defined, the technical terms used herein have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms herein is only for describing specific embodiments and is not intended to limit this application.

[0065] In the description of this application, the term "and / or" is used to describe a logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and both A and B exist. In addition, the character " / " in this document generally indicates that the objects before and after are in a logical "or" relationship.

[0066] In the present application, the terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantitative, primary or secondary or sequential relationship between the entities or operations.

[0067] In the present application, the "includes", "contains", "has" or other similar expressions used in the statements are intended to cover non-exclusive inclusion, and these expressions do not exclude the presence of additional elements in the process, method or product comprising the elements, so that the process, method or product comprising a series of elements can not only include those defined elements, but also include other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0068] As understood in the same way as in the "Guidelines for Examination", in the present application, the expressions "greater than", "less than", "exceed" and the like are understood as not including the number; the expressions "above", "below", "within" and the like are understood as including the number. In addition, in the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), and similar expressions related to "multiple" are also understood in this way, for example, "multiple groups", "multiple times" and the like, unless otherwise explicitly specified.

[0069] Please refer to Figure 1 In the first aspect, the present application provides a visual location recognition method for cross-modal retrieval, comprising the following steps:

[0070] S1: When constructing a 3D point cloud map database in an offline manner, compressing dense point clouds into a first lightweight structured point cloud;

[0071] S2: Inputting the first lightweight structured point cloud into a rotation learning network to generate a second lightweight structured point cloud;

[0072] S3: Inputting the second lightweight structured point cloud into an attention feature processing module, and using an attention mechanism to fuse context information for the second lightweight structured point cloud through the attention feature processing module to obtain a third lightweight structured point cloud with attention weight;

[0073] S4: Aggregating the third lightweight structured point cloud through a pooling layer to obtain a global feature of the 3D point cloud map database;

[0074] S5: Obtaining a to-be-queried image in an online manner, and extracting a global feature of the to-be-queried image through a lightweight image backbone network, inputting the global feature of the to-be-queried image into a pooling layer to aggregate into a 2D image global feature;

[0075] S6: Calculate the Euclidean distance between the global features of the 2D image and the global features of the 3D point cloud map database, perform feature matching between the two, and establish a corresponding relationship between the 2D image and the 3D point cloud.

[0076] In this embodiment, a lightweight image backbone network refers to a backbone network with an output dimension less than or equal to 256, including but not limited to a RepVGG-lite network, which combines the advantages of a residual network ResNet and a visual geometry group network VGG, and uses identity and 1×1 convolution branches based on 3×3 convolution and ReLU layers, but each layer takes the output of its only previous layer as input and feeds the output to its only next layer without skip transmission. RepVGG-lite is a lightweight RepVGG network with the least number of layers and channels. RepVGG-lite is more discriminative than a general network in visual position recognition tasks, has a faster feature extraction speed, and retains more local information of the image to improve feature matching capabilities.

[0077] For an image, each pixel cannot exist in isolation. Each pixel has a certain connection with the surrounding pixels. The mutual connection of a large number of pixels produces various objects in the image. The context information of an image refers to a certain connection between the pixels on the image and the surrounding pixels. Similarly, in this embodiment, for the point cloud structure, the context information refers to the relationship between a point cloud and the surrounding point clouds. By fusing the context information, more meaningful features can be extracted. The fusion of context information can be completed by the attention feature processing module, which obtains the weight corresponding to the attention through a series of calculations.

[0078] The above solution constructs a 3D point cloud map database in an offline manner at one time and converts it into a lightweight structure. While retaining the geometric features of the point cloud, it reduces the computing cost and obtains the image to be queried in real time online. The global features of the query image are extracted through a lightweight backbone network, and the point cloud database is matched cross-modally to retrieve the point cloud map corresponding to the query image. It can quickly extract the feature information of the query image and retain more local information, thereby improving the recognition accuracy and retrieval speed of the model and saving the installation cost of the vehicle-mounted lidar equipment.

[0079] In some embodiments, the first lightweight structure point cloud includes a point cloud in a normal distribution transform (NDT) format; and compressing the dense point cloud into the first lightweight structure point cloud includes compressing the dense point cloud into the point cloud in the normal distribution transform (NDT) format according to the following formula:

[0080]

[0081] Among them, μ represents the mathematical expectation, σ represents the variance, n represents the number of 3D points for normal distribution transformation, X k represents the kth 3D point transformed under normal distribution.

[0082] The point cloud NDT representation is a compact "spherical" structure. The first lightweight structure point cloud is a point cloud in a lightweight structure form. By compressing the dense point cloud into a point cloud in a lightweight structure form, it can not only maximize the retention of the geometric features of the point cloud, but also significantly reduce the computational cost.

[0083] In some embodiments, in step S2, the rotation learning network generates a second lightweight structure point cloud according to the following formula:

[0084]

[0085] Among them, T 3×3 Represents a 3×3 rotation matrix generated by the rotation learning network.

[0086] By inputting the compressed lightweight structure point cloud (i.e., the first lightweight structure point cloud) into a rotation learning network, the point cloud rotation invariance can be achieved.

[0087] In some embodiments, the attention feature processing module includes a plurality of attention encoders connected in series, and a shared fully connected layer with a jump connection is provided at the head and tail of the attention feature processing module; each of the attention encoders is configured to: use a multi-head self-attention layer to calculate weights, obtain 3D points with normal distribution transformations of attention weights, and input the obtained 3D points with normal distribution transformations into a feedforward network, and then perform normalization processing. Through processing by the attention feature processing module, a point cloud in the form of a lightweight structure with attention weights can be obtained. The structural diagram of the attention feature processing module is shown in FIG. Figure 3 shown.

[0088] like Figure 2 As shown, in some embodiments, step S4 may be followed by step S41: inputting the global features of the 3D point cloud map database obtained in step S4 into a multilayer perceptron for processing to obtain 3D point cloud global features of consistent size.

[0089] The lightweight point cloud with attention weights is aggregated through the pooling layer to obtain the global features of the 3D point cloud map database. The pooling layer includes but is not limited to the local aggregation feature vector network layer NetVLAD (Net Vector of Locally Aggregated Descriptors). The obtained point cloud global features are processed by a multilayer perceptron MLP (Multilayer Perceptron) to obtain the global features of the 3D point cloud map database with consistent size, which is convenient for subsequent further processing and calculation.

[0090] In certain embodiments, the method comprises:

[0091] A loss function is formed based on the positive and negative samples and the randomly sampled samples in the 2D image and 3D point cloud map database, and the parameters of the visual position recognition network are updated by the loss function;

[0092] The updating of the visual position recognition network parameters includes:

[0093] performing parameter updating on the rotation learning network;

[0094] and / or updating parameters of the attention feature processing module;

[0095] and / or updating parameters of the lightweight image backbone network;

[0096] And / or update parameters of the pooling layer.

[0097] Preferably, the loss function is a quadruple loss function, and the formula of the quadruple loss function is as follows:

[0098]

[0099] Among them, ξ q represents the global features of the query image, ξ p represents the global features of the positive samples in the point cloud database, ξ n Represents the global features of negative samples in the point cloud database, represents the global features randomly sampled in the query image and point cloud database, and is related to ξ q ,ξ p and ξ n are all different, d() represents the Euclidean distance, H() represents the hinge loss function, and α and β represent margin constants.

[0100] The model parameters are updated by calculating the quadruple loss function. The model loss comes from the global feature learning of the 2D query image and the global feature learning of the positive and negative samples in the 3D point cloud database. The quadruple loss function aims to shorten the distance between the query image and the positive sample point cloud, increase the distance between the query image and the negative sample point cloud, and increase the distance between the query image and the remaining sample point cloud, which can make visual position recognition more accurate.

[0101] In certain embodiments, the method comprises:

[0102] Retrieving a point cloud map corresponding to the image to be queried from the 3D point cloud map database;

[0103] And the global features of the image to be queried and the global features of the positive and negative samples in the 3D point cloud map database are trained and learned to improve the accuracy of the 3D point cloud map database retrieved in real time based on the image to be queried; the image to be queried is a 2D image.

[0104] By calculating the Euclidean distance between the global features of the 2D image and the global features of the 3D point cloud, feature matching is directly performed and the correspondence between the image and the point cloud is established. Then, the point cloud map corresponding to the 2D query image can be retrieved from the 3D point cloud map database to achieve efficient visual position recognition from 2D image to 3D point cloud retrieval.

[0105] In a second aspect, the present invention further provides a storage medium storing a computer program, which, when executed, implements the method described in the first aspect of the present invention.

[0106] In the third aspect, Figure 6 As shown, the present invention also provides an electronic device 10, including a storage medium 102 and a processor 101, the storage medium 102 is the storage medium as described in the second aspect; the processor 101 is electrically connected to the storage medium 102, and is used to execute the computer program stored in the storage medium to implement the method as described in the first aspect.

[0107] Preferably, the electronic device may be a computer device, including but not limited to personal computers, servers, general-purpose computers, special-purpose computers, network devices, embedded devices, programmable devices, smart mobile terminals, smart home devices, wearable smart devices, and in-vehicle smart devices. Storage media include but are not limited to RAM, ROM, magnetic disks, tapes, optical disks, flash memory, USB flash drives, mobile hard drives, memory cards, memory sticks, network server storage, and network cloud storage. Processors include but are not limited to CPUs (central processing units), GPUs (graphics processing units), and MCUs (microprocessors).

[0108] like Figure 5As shown, in a fourth aspect, the present invention further proposes a visual position recognition device for cross-modal retrieval from 2D images to 3D point clouds, the device comprising:

[0109] Conversion module 1 builds a 3D point cloud map database in an offline manner at one time, converting the dense point cloud representation in the database into a point cloud in a lightweight structure form, and the lightweight point cloud is processed by the rotation learning network;

[0110] A first sending module 2 is used to send the point cloud in the lightweight structure form to the attention feature processing module;

[0111] A second sending module 3, configured to send a 2D query image to the lightweight image backbone network;

[0112] The first generation module 4 uses the attention mechanism to fuse context information for the lightweight point cloud to obtain a lightweight point cloud with attention weights, and generates the global features of the map database through the pooling layer;

[0113] The second generation module 5 obtains the query image in real time online, extracts features with more local information through a lightweight image backbone network, queries the global features of the query image, and generates the global features of the query image through a pooling layer;

[0114] Learning module 6 is used for global feature learning of 2D query images and global feature learning of positive and negative samples in the 3D point cloud map database to improve the accuracy of real-time retrieval of 3D point cloud map database using 2D query images;

[0115] The updating module 7 is used to form a four-tuple loss function according to the positive and negative samples and random sampling samples in the 2D query image and the 3D point cloud map database, and update the visual position recognition network parameters by calculating the loss function.

[0116] like Figure 4 As shown, the present invention proposes a visual position recognition method for cross-modal retrieval from 2D images to 3D point clouds, which constructs a 3D point cloud map database in an offline manner at one time and converts it into a lightweight structure. While retaining the geometric features of the point cloud, it reduces the computing cost, obtains the image to be queried in real time online, extracts the global features of the query image through a lightweight backbone network, and matches the point cloud database cross-modally to retrieve the point cloud map corresponding to the query image. It can quickly extract the feature information of the query image and retain more local information, thereby improving the recognition accuracy and retrieval speed of the model and saving the installation cost of the vehicle-mounted lidar equipment.

[0117] Although the above embodiments have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concepts. Therefore, the above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present invention.

Claims

1. A visual location recognition method for cross-modal retrieval, characterized by: The method comprises the following steps: S1: When constructing a 3D point cloud map database in an offline manner, the dense point cloud is compressed into the first lightweight structure point cloud; S2: inputting the first lightweight structure point cloud into a rotation learning network to generate a second lightweight structure point cloud; S3: Inputting the second lightweight structure point cloud into an attention feature processing module, and using the attention feature processing module to fuse context information for the second lightweight structure point cloud using an attention mechanism to obtain a third lightweight structure point cloud with attention weights; S4: aggregating the third lightweight structure point cloud through a pooling layer to obtain global features of the 3D point cloud map database; S5: obtaining a query image online, extracting global features of the query image through a lightweight image backbone network, and inputting the global features of the query image into a pooling layer to aggregate them into 2D image global features; S6: Calculating the Euclidean distance between the global features of the 2D image and the global features of the 3D point cloud map database, performing feature matching between the two, and establishing a correspondence between the 2D image and the 3D point cloud; The method comprises: A loss function is formed based on the positive and negative samples and the randomly sampled samples in the 2D image and 3D point cloud map database, and the parameters of the visual position recognition network are updated by the loss function; The updating of the visual position recognition network parameters includes: Updating parameters of the rotation learning network; and / or updating parameters of the attention feature processing module; and / or updating parameters of the lightweight image backbone network; and / or updating parameters of the pooling layer; The loss function is a quadruple loss function, and the formula of the quadruple loss function is as follows: Among them, ξ q represents the global features of the query image, ξ p represents the global features of the positive samples in the point cloud database, ξ n Represents the global features of negative samples in the point cloud database, represents the global features randomly sampled in the query image and point cloud database, and is related to ξ q ,ξ p and ξ n are all different, d() represents the Euclidean distance, H() represents the hinge loss function, and α and β represent margin constants.

2. The visual location recognition method for cross-modal retrieval according to claim 1, wherein: The first lightweight structure point cloud includes a point cloud in a normal distribution transform (NDT) form; The compressing the dense point cloud into a first lightweight structure point cloud includes: The dense point cloud is compressed into a point cloud in the normal distribution transform (NDT) form according to the following formula: Among them, μ represents the mathematical expectation, σ represents the variance, n represents the number of 3D points for normal distribution transformation, X k represents the kth 3D point transformed under normal distribution.

3. The visual location recognition method for cross-modal retrieval according to claim 1, wherein: In step S2, the rotation learning network generates a second lightweight structure point cloud according to the following formula: Among them, T 3×3 Represents a 3×3 rotation matrix generated by the rotation learning network.

4. The visual location recognition method for cross-modal retrieval according to claim 1, wherein: The attention feature processing module includes a plurality of attention encoders connected in series, and a shared fully connected layer with skip connections at the head and tail of the attention feature processing module; Each of the attention encoders is configured to: calculate weights using a multi-head self-attention layer to obtain 3D points transformed by a normal distribution with attention weights, and input the obtained 3D points transformed by a normal distribution into a feedforward network, and then perform normalization processing.

5. The visual location recognition method for cross-modal retrieval according to claim 1, wherein: Step S4 includes step S41: inputting the global features of the 3D point cloud map database obtained in step S4 into a multi-layer perceptron for processing to obtain 3D point cloud global features of consistent size.

6. The visual location recognition method for cross-modal retrieval according to claim 1, wherein: The method comprises: Retrieving a point cloud map corresponding to the image to be queried from the 3D point cloud map database; And the global features of the image to be queried and the global features of the positive and negative samples in the 3D point cloud map database are trained and learned to improve the accuracy of the 3D point cloud map database retrieved in real time based on the image to be queried; the image to be queried is a 2D image.

7. A storage medium, characterized in that: The storage medium stores a computer program, which implements the method according to any one of claims 1 to 6 when executed.

8. An electronic device, characterized in that: include: The storage medium is the storage medium according to claim 7; A processor is electrically connected to the storage medium, and is configured to execute a computer program stored in the storage medium to implement the method according to any one of claims 1 to 6.

Images