Multimodal visual positioning method, apparatus, and electronic device

Abstract

This application provides a multimodal visual localization method, apparatus, and electronic device, which can be applied to the field of computer vision technology. The method includes: responding to image data acquired by an image acquisition device mounted on a vehicle at a target time, extracting features from the image data to obtain image features; predicting the depth features at the target time based on the expected pose transformation of the image acquisition device and historical depth features to obtain predicted depth features; if the image data also includes a depth image, performing noise reduction processing on the predicted depth features according to the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features; fusing the optimized depth features and visual features to obtain multimodal visual features; and localizing the image acquisition device based on the multimodal visual features to obtain the localization result at the target time.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically to a multimodal visual positioning method, apparatus, and electronic device. Background Technology

[0002] In scenarios such as autonomous driving and vehicle navigation, image acquisition device positioning based on multimodal visual features is the core technology for achieving accurate vehicle positioning. Multimodal positioning, which integrates visual image texture information and depth image spatial depth information, can comprehensively perceive the vehicle's surrounding environment. Its positioning accuracy directly determines the reliability of vehicle environmental perception and path planning, and is a key link in ensuring the safe operation of autonomous driving.

[0003] However, since depth features rely entirely on real-time acquired depth images, in tunnel environments, limitations such as lighting conditions can lead to missing or distorted depth images in some frames. Under these circumstances, it is difficult to obtain effective depth features, resulting in large deviations in positioning results and making it difficult to meet the high-precision positioning requirements of vehicles. Summary of the Invention

[0004] In view of the above problems, this application provides a multimodal visual positioning method, apparatus and electronic device.

[0005] According to a first aspect of this application, a multimodal visual localization method is provided, comprising: responding to an image acquisition device mounted on a vehicle acquiring image data at a target time, extracting features from the image data to obtain image features, wherein the image data includes a visual image and the image features include visual features; predicting the depth features at the target time based on the expected pose change of the image acquisition device and historical depth features to obtain predicted depth features, wherein the historical depth features are obtained by extracting features from historical depth images, wherein the historical depth images are depth images acquired by the image acquisition device at historical times, and the expected pose change is predicted based on the historical pose change of the image acquisition device; if it is determined that the image data also includes a depth image, performing noise reduction processing on the predicted depth features according to the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features, wherein the depth features are obtained by extracting features from the depth image; fusing the optimized depth features and visual features to obtain multimodal visual features; and localizing the image acquisition device based on the multimodal visual features to obtain a localization result at the target time.

[0006] According to an embodiment of this application, the depth features at a target time are predicted based on the expected pose change of the image acquisition device and historical depth features, thereby obtaining predicted depth features. This includes: determining the motion trajectory of the image acquisition device from a historical time to a target time based on the expected pose change; and predicting the depth features at the target time based on historical depth features and the motion trajectory, thereby obtaining predicted depth features.

[0007] According to an embodiment of this application, the multimodal visual localization method further includes: determining a covariance matrix for characterizing the confidence of predicted depth features based on a state transition matrix, a historical covariance matrix, and a noise covariance matrix for characterizing the motion of the image acquisition device, wherein the historical covariance matrix is ​​the covariance matrix at a historical moment.

[0008] According to an embodiment of this application, the predicted depth features are denoised based on the mapping relationship between depth features and predicted depth features to obtain optimized depth features. This includes: determining a measurement matrix based on the mapping relationship, the measurement matrix being used to describe the degree of influence of changes in depth features on predicted depth features; determining a Kalman gain based on the measurement matrix and the covariance matrix; and denoising the predicted depth features based on the Kalman gain and the measurement noise to obtain optimized depth features, where the measurement noise is the residual between the depth features and the predicted depth features.

[0009] According to an embodiment of this application, the multimodal visual localization method further includes: when it is determined that the image data does not include a depth image, performing feature fusion on the predicted depth features and visual features to obtain multimodal visual features.

[0010] According to embodiments of this application, the multimodal visual localization method further includes: performing feature matching between multimodal visual features and historical multimodal visual features to determine the correspondence between feature points in the multimodal visual features and historical multimodal visual features, wherein the historical multimodal visual features represent multimodal visual features at historical moments; determining a preset number of feature point groups from the multimodal visual features and historical multimodal visual features based on the correspondence, wherein each feature point group includes a feature point in the multimodal visual features and a historical feature point in the historical multimodal visual features corresponding to the feature point; determining the rotation matrix and translation vector of the image acquisition device based on the two-dimensional coordinates of the feature points, the three-dimensional coordinates of the historical feature points, and the intrinsic parameters of the image acquisition device; and determining the pose transformation of the image acquisition device based on the rotation matrix and translation vector.

[0011] According to an embodiment of this application, the image acquisition device is located based on multimodal visual features to obtain the location result at the target time, including: determining the location result at the target time based on the pose transformation and the historical location results of the image acquisition device at historical times.

[0012] According to embodiments of this application, the feature fusion method for optimizing deep features and visual features includes at least one of the following: feature concatenation, feature weighted summation.

[0013] The second aspect of this application provides a multimodal visual positioning device, comprising: a feature extraction module, configured to extract features from image data acquired by an image acquisition device mounted on a vehicle at a target time to obtain image features, wherein the image data includes a visual image and the image features include visual features; a feature prediction module, configured to predict depth features at the target time based on the expected pose change of the image acquisition device and historical depth features to obtain predicted depth features, wherein the historical depth features are obtained by feature extraction from historical depth images, which are depth images acquired by the image acquisition device at historical times, and the expected pose change is predicted based on the historical pose change of the image acquisition device; a feature denoising module, configured to perform denoising processing on the predicted depth features based on the mapping relationship between the depth features and the predicted depth features when it is determined that the image data also includes a depth image, to obtain optimized depth features, wherein the depth features are obtained by feature extraction from the depth image; a feature fusion module, configured to fuse the optimized depth features and visual features to obtain multimodal visual features; and a device positioning module, configured to locate the image acquisition device based on the multimodal visual features to obtain the positioning result at the target time.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to embodiments of this application, a processing flow of depth feature prediction, noise reduction optimization, and multimodal fusion can achieve high accuracy and robustness in multimodal visual positioning. Predicting depth features based on device pose transformation solves the problem of no effective depth features when depth images are missing. Furthermore, noise reduction optimization can be performed before real-time depth feature fusion to reduce noise interference with depth features, resulting in higher accuracy of the fused multimodal features and ultimately improving the accuracy and stability of the positioning results. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0019] Figure 1 The diagram illustrates application scenarios of the multimodal visual positioning method, apparatus, and electronic device according to embodiments of this application.

[0020] Figure 2 A flowchart of a multimodal visual localization method according to an embodiment of this application is shown.

[0021] Figure 3 A data flow diagram of a multimodal visual localization method according to an embodiment of this application is shown.

[0022] Figure 4 A structural block diagram of a multimodal visual positioning device according to an embodiment of this application is shown.

[0023] Figure 5 A block diagram of an electronic device suitable for implementing a multimodal visual positioning method according to an embodiment of this application is shown. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0029] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0030] Embodiments of this application provide a multimodal visual localization method, comprising: responding to an image acquisition device mounted on a vehicle acquiring image data at a target time, extracting features from the image data to obtain image features, wherein the image data includes a visual image and the image features include visual features; predicting the depth features at the target time based on the expected pose change of the image acquisition device and historical depth features to obtain predicted depth features, wherein the historical depth features are obtained by extracting features from historical depth images, wherein the historical depth images are depth images acquired by the image acquisition device at historical times, and the expected pose change is predicted based on the historical pose change of the image acquisition device; when it is determined that the image data also includes a depth image, performing noise reduction processing on the predicted depth features according to the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features, wherein the depth features are obtained by extracting features from the depth image; fusing the optimized depth features and visual features to obtain multimodal visual features; and localizing the image acquisition device based on the multimodal visual features to obtain the localization result at the target time.

[0031] Figure 1 The diagram illustrates application scenarios of the multimodal visual positioning method, apparatus, and electronic device according to embodiments of this application.

[0032] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, an image acquisition device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the image acquisition device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0033] Users can use the first terminal device 101, the second terminal device 102, and the image acquisition device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Communication client applications, such as map applications, can be installed on the first terminal device 101 and the second terminal device 102.

[0034] The image acquisition device 103 can be any type of vehicle-mounted device used for acquiring images, such as vehicle-mounted cameras, reversing cameras, etc., including but not limited to devices used for acquiring RGB images, depth images, and pseudo-color images.

[0035] The first terminal device 101 and the second terminal device 102 can be various electronic devices with displays and support web browsing, including but not limited to in-vehicle smart devices, smartphones, tablets, laptops, and desktop computers.

[0036] Server 105 can be a server that provides various services, such as a background management server that supports applications browsed by users using the first terminal device 101 and the second terminal device 102 (this is just an example). The background management server can analyze and process the image data submitted by the image acquisition device 103 and feed the processing results back to the terminal devices.

[0037] It should be noted that the multimodal visual positioning method provided in this application embodiment can generally be executed by server 105. Correspondingly, the multimodal visual positioning device provided in this application embodiment can generally be located in server 105. The multimodal visual positioning method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the image acquisition device 103, and / or server 105. Correspondingly, the multimodal visual positioning device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the image acquisition device 103, and / or server 105.

[0038] It should be understood that Figure 1 The number of first terminal devices, second terminal devices, third terminal devices, networks, and servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of first terminal devices, second terminal devices, third terminal devices, networks, and servers can be included.

[0039] The following will be based on Figure 1 The described scene, through Figures 2-3 The multimodal visual positioning method according to the embodiments of this application will be described in detail.

[0040] Figure 2 A flowchart of a multimodal visual localization method according to an embodiment of this application is shown.

[0041] like Figure 2 As shown, the multimodal visual localization method of this embodiment includes operations S210 to S250.

[0042] In operation S210, in response to the image acquisition device mounted on the vehicle acquiring image data at the target time, feature extraction is performed on the image data to obtain image features.

[0043] In operation S220, based on the expected pose change of the image acquisition device and historical depth features, the depth features at the target time are predicted to obtain the predicted depth features.

[0044] In operation S230, if it is determined that the image data also includes a depth image, the predicted depth features are denoised according to the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features.

[0045] In operation S240, the optimized depth features and visual features are fused to obtain multimodal visual features.

[0046] When operating the S250, the image acquisition device is located based on multimodal visual features to obtain the location result of the target at that time.

[0047] Image acquisition devices can be used to acquire image data, which may include visual images. Visual images can be full-color images, such as RGB images.

[0048] Image features can be obtained by extracting features from image data. These image features can include visual features, which are obtained by extracting features from visual images.

[0049] Taking RGB images as an example, when extracting features from visual images, the Oriented FAST and Rotated BRIEF (ORB) features of RGB images can be extracted to obtain the key points of the RGB images and the binary descriptors of each key point, which are used to represent the visual appearance information of the RGB images.

[0050] In one example, a preset number of descriptors can be selected for concatenation, and the concatenated vector can be normalized to obtain ORB features.

[0051] In another example, the bag-of-words model can be used to generate feature vectors. When constructing the dictionary of the bag-of-words model, a preset number of RGB images are randomly selected from the dataset, the descriptors of the images are extracted and clustered, and the descriptors of the RGB images to be encoded are assigned to appropriate cluster centers. The occurrence frequency of each cluster center is counted as the feature vector.

[0052] Historical depth images can be obtained from the historical behavior database collected by the image acquisition device, and historical depth features can be determined. The historical depth images are depth images collected by the image acquisition device at historical moments, and the historical depth features are obtained by extracting features from the historical depth images.

[0053] The historical pose changes of the image acquisition device can be obtained from the historical behavior database, and the expected pose changes of the image acquisition device can be predicted based on the historical pose changes of the image acquisition device.

[0054] In one example, the historical pose transformation represents the pose transformation of the image acquisition device between a first historical moment and a second historical moment, where the first historical moment is earlier than the second historical moment. The expected pose transformation represents the prediction of the pose transformation of the image acquisition device between the second historical moment and the target moment. The second historical moment is the same as the acquisition time of the historical depth image.

[0055] The desired pose transformation can characterize the motion state of the image acquisition device from the historical moment to the target moment. Based on this motion state and historical depth features, the depth features at the target moment can be predicted to obtain the predicted depth features.

[0056] Image acquisition devices can also be used to acquire depth images, which can be images representing the distance between objects in the image and the image acquisition device. Depth features are obtained by extracting features from depth images.

[0057] When acquiring image data using an image acquisition device at a target time, both visual and depth images at that target time can be obtained simultaneously.

[0058] In the process of feature extraction from depth images, structured features can be extracted. The depth map is divided into a regular grid, and statistical information such as the average depth within each grid is calculated to construct a feature vector for the depth features. This depth feature extraction process can extract features from depth images that reflect scene structure and distance information.

[0059] The first physical module used to acquire visual images in an image acquisition device differs from the second physical module used to acquire depth images. The different working principles and data processing complexities of the first and second physical modules result in differences in maximum frame rate and stable operating frame rate. In scenarios with limited lighting and positioning signals, such as tunnels, the second physical module may experience a decrease in the effective frame rate or even data loss due to insufficient lighting and light reflection.

[0060] When an image acquisition device acquires a depth image, i.e., the image data includes a depth image, features can be extracted from the depth image to obtain depth features. Then, the depth features of the observed depth image can be used to reduce noise in the predicted depth features to obtain optimized depth features.

[0061] Specifically, the mapping relationship between deep features and predicted deep features can be determined. This mapping relationship represents the correspondence between feature points in the deep features and the predicted deep features. Furthermore, based on the mapping relationship between the deep features and the predicted deep features, noise reduction processing can be performed on the feature points in the predicted deep features to obtain optimized deep features.

[0062] After obtaining the optimized depth features, the optimized depth features and visual features can be fused to obtain multimodal visual features. Based on the multimodal visual features, the image acquisition device can be located to obtain the location result of the target time.

[0063] For example, a latent space can be constructed based on the bag-of-words model. A word dictionary is used to map visual features and optimized deep features into low-dimensional semantic feature vectors, which are then fused. The latent space refers to the potential, not easily observed, feature space within the data. In multimodal data, different modalities may exhibit inherent relationships and structures, which can be represented by the latent space. The purpose of latent space modeling is to uncover implicit information in the data and improve system performance. In this example, the bag-of-visual-words (BoVW) model can be used for latent space modeling. The BoVW model clusters feature descriptors in an image to obtain a set of visual words, and then represents the image as a histogram of visual words. By constructing the BoVW bag-of-words model, RGB and deep images can be converted into low-dimensional feature vectors, thereby achieving latent space modeling.

[0064] Multimodal visual features can be used to determine the pose transformation and movement of the image acquisition device from a historical moment to a target moment. Based on the localization results, pose transformation, and movement at the historical moment, the localization result at the target moment can be determined. Since the image acquisition device is usually configured at a fixed position on the vehicle, the localization result of the vehicle at the target moment can be obtained based on the localization result of the image acquisition device at the target moment.

[0065] According to embodiments of this application, a process of predicting depth features, noise reduction optimization, and multimodal fusion can achieve high accuracy and robustness in multimodal visual positioning. Predicting depth features based on device pose transformation solves the problem of no effective depth features when depth images are missing. Furthermore, noise reduction optimization can be performed before real-time depth feature fusion to reduce noise interference with depth features, resulting in higher accuracy of the fused multimodal features and ultimately improving the accuracy and stability of the positioning results.

[0066] According to an embodiment of this application, the depth features at a target time are predicted based on the expected pose change of the image acquisition device and historical depth features, thereby obtaining predicted depth features. This includes: determining the motion trajectory of the image acquisition device from a historical time to a target time based on the expected pose change; and predicting the depth features at the target time based on historical depth features and the motion trajectory, thereby obtaining predicted depth features.

[0067] The motion trajectory of the image acquisition device from the historical moment to the target moment can be determined based on the expected pose change at each moment from the historical moment to the target moment. The motion trajectory can include the translation and rotation of the image acquisition device.

[0068] Based on historical depth features and motion trajectory, the depth features of the target at any given time are predicted. The process of obtaining the predicted depth features is shown in Equation (1):

[0069] (1)

[0070] Where k represents the current time, and k-1 represents a historical time. This indicates that the prediction depth features are obtained by making predictions based on historical depth features from previous times. The historical depth characteristics of a historical moment are represented by f(·), which is the system model function.

[0071] According to the embodiments of this application, the motion trajectory of the device is derived based on the expected pose transformation, and the depth features at the target time are predicted by combining historical depth features. This makes the predicted depth features highly matched with the actual motion state of the device, avoids blind prediction without trajectory constraints, improves the accuracy of depth feature prediction, and provides high-quality basic data for subsequent noise reduction or direct fusion.

[0072] According to an embodiment of this application, the multimodal visual localization method further includes: determining a covariance matrix for characterizing the confidence of predicted depth features based on a state transition matrix, a historical covariance matrix, and a noise covariance matrix for characterizing the motion of the image acquisition device, wherein the historical covariance matrix is ​​the covariance matrix at a historical moment.

[0073] The state transition matrix can be determined based on the state equation determined during the initialization process, where the state transition method is shown in equation (2):

[0074] (2)

[0075] in, This represents the state vector at the current moment. Let F represent the state vector at a historical moment, F represent the state transition matrix, and B represent the control input matrix. Represents the process noise vector. Let represent the noise covariance matrix. The current state vector can be represented by the predicted depth features of the current time step, and the state vectors of historical time steps can be represented by the optimized depth features of the historical time steps.

[0076] Based on the state transition matrix, the historical covariance matrix, and the noise covariance matrix, the covariance matrix used to characterize the confidence level of the predicted depth features is determined, as shown in equation (3):

[0077] (3)

[0078] in, This represents the covariance matrix at the current moment, determined based on the historical covariance matrix at previous times. Let be the state transition matrix at the current moment. This represents the transpose of the state transition matrix at the current moment. The historical covariance matrix represents a historical moment. This represents the noise covariance matrix at the current moment.

[0079] According to embodiments of this application, the confidence level of device motion is quantified by a state transition matrix, and the confidence level of predicted depth features is represented by a covariance matrix, thereby achieving a probabilistic and quantitative description of the predicted depth features. This provides a crucial confidence level basis for subsequent Kalman denoising, making the denoising process more closely match the confidence level distribution of depth features and improving the targeting of denoising.

[0080] According to an embodiment of this application, the predicted depth features are denoised based on the mapping relationship between depth features and predicted depth features to obtain optimized depth features. This includes: determining a measurement matrix based on the mapping relationship, the measurement matrix being used to describe the degree of influence of changes in depth features on predicted depth features; determining a Kalman gain based on the measurement matrix and the covariance matrix; and denoising the predicted depth features based on the Kalman gain and the measurement noise to obtain optimized depth features, where the measurement noise is the residual between the depth features and the predicted depth features.

[0081] The mapping relationship between deep features and predicted deep features can be represented by equation (4):

[0082] (4)

[0083] in, This represents the measured value, i.e., the depth feature. Represents the measurement matrix. Let R represent the measurement noise, and let R represent the measurement noise covariance matrix.

[0084] The measurement matrix H can be determined using equation (4).

[0085] Determine the Kalman gain based on the measurement matrix and covariance matrix. The process is shown in equation (5):

[0086] (5)

[0087] in, The measurement matrix at the current moment. This represents the transpose of the measurement matrix at the current moment. This represents the measurement noise covariance matrix at the current moment.

[0088] Based on the mapping relationship shown in equation (4), the residual between the depth features and the predicted depth features, i.e., the measurement noise, can be determined. According to the Kalman gain and the measurement noise, the predicted depth features are denoised to obtain the optimized depth features. The process is shown in equation (6):

[0089] (6)

[0090] Where h(·) represents the measurement model function, used to predict deep features in the latent space. This is mapped to a theoretical predicted value. In this embodiment, the measurement model function is derived from the measurement matrix. This implementation is used to extract the actual observed depth feature dimensions from the complete state vector to compensate for missing data.

[0091] To ensure that the covariance matrix of the target time can be successfully obtained when locating based on the image data after subsequent image data, the covariance matrix can be updated. The specific update process is shown in equation (7):

[0092] (7)

[0093] Where I is the identity matrix, This is the updated covariance matrix.

[0094] In addition, the Kalman gain can be updated as shown in equation (8):

[0095] (8)

[0096] in, This represents the updated Kalman gain. Through the above measurement, prediction, and update process, depth images and depth features can be predicted and corrected in the feature latent space to obtain more accurate optimized depth features.

[0097] According to the embodiments of this application, a measurement matrix is ​​constructed based on the mapping relationship between depth features and predicted depth features, and the Kalman gain is calculated in combination with the covariance matrix. The Kalman gain is used to optimize the noise reduction of the predicted depth features, thereby achieving accurate noise reduction of depth features, effectively filtering out noise interference in depth features, and significantly improving the quality of depth features.

[0098] According to an embodiment of this application, the multimodal visual localization method further includes: when it is determined that the image data does not include a depth image, performing feature fusion on the predicted depth features and visual features to obtain multimodal visual features.

[0099] The second physical module in the image acquisition device may also fail to acquire depth images of certain frames due to insufficient lighting or light reflection.

[0100] When the depth image at the target time is missing, the depth features at the target time cannot be obtained, and therefore the predicted depth features cannot be denoised and optimized based on the depth features. Therefore, the predicted depth features and visual features can be directly fused to obtain multimodal visual features.

[0101] According to the embodiments of this application, when the depth image is missing, the predicted depth features and visual features are directly fused to ensure the continuity of multimodal visual feature generation, avoid the interruption of the localization process due to the lack of depth images, greatly improve the robustness of the localization method, and adapt to the localization needs in complex scenarios.

[0102] According to embodiments of this application, the feature fusion method for optimizing deep features and visual features includes at least one of the following: feature concatenation, feature weighted summation.

[0103] The feature splicing method is shown in equation (9):

[0104] (9)

[0105] in, For multimodal visual features, Indicates visual characteristics, [·] indicates optimization of deep features, and [·] indicates vector concatenation.

[0106] The feature fusion method is shown in equation (10):

[0107] (10)

[0108] Where α and β are weighting coefficients, and α+β=1.

[0109] It is understandable that the feature fusion method that fuses predicted deep features and visual features can be the same as the feature fusion method that fuses optimized deep features and visual features.

[0110] After feature fusion is completed, the multimodal visual features can be standardized. For example, the mean normalization and variance scaling of the multimodal visual features can be performed using the data standardization method (StandardScaler) so that the mean of each feature component is 0 and the variance is 1, thereby improving the stability and comparability of the multimodal visual features.

[0111] According to embodiments of this application, various feature fusion methods, such as feature stitching and weighted summation, can be used to adapt to feature fusion needs in different scenarios, enhancing the flexibility of localization. Different fusion methods can specifically strengthen the weights of visual and depth features, improve the expressive power of multimodal features, and further optimize the localization effect.

[0112] According to embodiments of this application, the multimodal visual localization method further includes: performing feature matching between multimodal visual features and historical multimodal visual features to determine the correspondence between feature points in the multimodal visual features and historical multimodal visual features, wherein the historical multimodal visual features represent multimodal visual features at historical moments; determining a preset number of feature point groups from the multimodal visual features and historical multimodal visual features based on the correspondence, wherein each feature point group includes a feature point in the multimodal visual features and a historical feature point in the historical multimodal visual features corresponding to the feature point; determining the rotation matrix and translation vector of the image acquisition device based on the two-dimensional coordinates of the feature points, the three-dimensional coordinates of the historical feature points, and the intrinsic parameters of the image acquisition device; and determining the pose transformation of the image acquisition device based on the rotation matrix and translation vector.

[0113] By feature matching, the correspondence between feature points in multimodal visual features and historical multimodal visual features can be determined. Based on the above correspondence, multiple feature point groups can be determined, and the positions of feature points in each feature point group in multimodal visual features and the positions of historical feature points in historical multimodal visual features can be determined.

[0114] The position of a feature point in a multimodal visual feature can be represented by two-dimensional coordinates, while the position of a historical feature point in a historical multimodal visual feature can be represented by three-dimensional coordinates.

[0115] Based on the feature points and historical feature points in each feature point group, the pose transformation of the image acquisition device can be determined through pose calculation methods.

[0116] The pose calculation method can employ the Perspective-n-Point (PnP) method. The PnP algorithm is a classic algorithm for solving camera pose. It calculates the camera's rotation and translation matrices using known 3D points and their corresponding 2D image points. When using the PnP algorithm to determine the pose transformation of the image acquisition device, the preset number of feature point groups can be set to greater than or equal to 4. This is because the PnP algorithm's solution process is essentially a nonlinear optimization problem, and 4 feature point groups provide sufficient constraints to solve for the 6 degrees of freedom of the rotation and translation matrices, including 3 rotational degrees of freedom and 3 translational degrees of freedom. Preferably, the preset number can be 10.

[0117] Based on the two-dimensional coordinates of the feature points, the three-dimensional coordinates of the historical feature points, and the intrinsic parameters of the image acquisition device, the rotation matrix and translation vector of the image acquisition device can be determined, as shown in equation (11):

[0118] (11)

[0119] in, Represents the depth scale factor. Let represent the two-dimensional coordinates of the feature point, K represent the intrinsic parameters of the image acquisition device, R' represent the rotation matrix of the image acquisition device, and t represent the translation vector of the image acquisition device. The three-dimensional coordinates of historical feature points.

[0120] In solving the above problem, the three-dimensional coordinates of historical feature points can be represented as a weighted sum using control points, as shown in equation (12):

[0121] (12)

[0122] in, Indicates weight, =1, This represents the three-dimensional coordinates of the j-th control point in the world coordinate system.

[0123] After establishing the system of equations based on equations (11) and (12), the system of equations can be solved by singular value decomposition.

[0124] In addition, the Random Sample Consensus (RANSAC) algorithm can be introduced to eliminate mismatches and enhance the robustness of the algorithm. The number of iterations N of the RANSAC algorithm can be determined by equation (13):

[0125] (13)

[0126] Where p is the confidence level, w is the inlier rate, and n is the number of samplings.

[0127] For cases using only monocular image sequences, the pose change of the image acquisition device can be estimated by calculating the essential matrix E between historical multimodal visual features and multimodal visual features. The essential matrix satisfies the epipolar geometric constraint shown in equation (14):

[0128] (14)

[0129] in, Representing feature points in multimodal visual features, This represents the transpose of feature points in multimodal visual features. This represents historical feature points in historical multimodal visual features. and These are feature points within the same feature point group, and E represents the essential matrix.

[0130] Furthermore, the essential matrix E can be solved using the eight-point method, and the relative pose can be obtained by matrix decomposition of the essential matrix E, thereby determining the pose transformation of the image acquisition device.

[0131] According to the embodiments of this application, by matching multimodal visual features with historical multimodal visual features, the correspondence of feature points is determined and the rotation matrix and translation vector are solved, so as to accurately quantify the device pose transformation and provide accurate pose basis for the calculation of positioning results, thereby further improving the positioning accuracy from the pose dimension.

[0132] According to an embodiment of this application, the image acquisition device is located based on multimodal visual features to obtain the location result at the target time, including: determining the location result at the target time based on the pose transformation and the historical location results of the image acquisition device at historical times.

[0133] Based on the pose transformation, the movement of the image acquisition device from the historical time to the target time can be determined. Based on the movement and the historical positioning results at the historical time, the position of the image acquisition device after moving according to the movement based on the historical positioning results can be determined, which is the positioning result at the target time.

[0134] Specifically, the rotation matrix and translation vector can be converted into a homogeneous transformation matrix, and the homogeneous transformation matrix can be multiplied with the historical positioning results to obtain the positioning result at the target time.

[0135] According to the embodiments of this application, the target time positioning result is derived based on pose transformation and historical positioning results, realizing the time-series iterative update of the positioning result, so that the positioning result fits the actual motion trajectory of the device, avoids the deviation of isolated frame positioning, and improves the continuity and accuracy of positioning.

[0136] Figure 3 A data flow diagram of a multimodal visual localization method according to an embodiment of this application is shown.

[0137] like Figure 3 As shown, after obtaining image data 301, feature extraction is performed on image data 301 to obtain image features 302. Image data 301 includes visual image 3011, and image features 302 include visual features 3021. Image data 301 may also include depth image 3012. When image data includes depth image 3012, image features 302 include depth features 3022.

[0138] The expected pose transformation 303 and historical depth features 304 can be obtained from the historical behavior database, and prediction can be made based on these to obtain the predicted depth features 305.

[0139] Given depth feature 3022, the predicted depth feature 305 is denoised based on depth feature 3022 to obtain optimized depth feature 306. The optimized depth feature 306 and visual feature 3021 are then fused to obtain multimodal visual feature 307.

[0140] In the absence of depth feature 3022, the predicted depth feature 305 and visual feature 3021 are fused to obtain multimodal visual feature 307.

[0141] Based on multimodal visual features 307, the image acquisition device can be located to obtain the location result 308 at the target time.

[0142] It should be noted that the multimodal visual localization method of this application can be validated using benchmark datasets in the field of multimodal perception. The benchmark dataset can be acquired simultaneously using a high-precision motion capture system, including RGB images, depth maps, and six-DOF pose ground truth values. It contains 49 independent sequences, covering four major categories of complex scenes: static scenes, dynamic disturbances, sudden changes in illumination, and low-texture environments. It is particularly suitable for algorithm validation in tunnel environments.

[0143] The data in the benchmark dataset can be preprocessed, and the data association and feature extraction of RGB images and depth maps can be aligned by timestamps. Nanosecond-level timestamps can be obtained by using the RGB images and depth maps attached to each frame, and cross-modal data can be associated to ensure that the data meets the temporal consistency requirements of subsequent dynamic scenes.

[0144] In practical applications, RGB sensor data is often acquired at a high frequency, while depth sensor data is often acquired at a low frequency, and some sensor data may contain redundant information. Therefore, frame extraction can be used to reduce the amount of data, simulate data loss in real-world applications, and improve computational efficiency. Depth images can be randomly selected and discarded according to a preset frame extraction rate.

[0145] In the embodiments of this application, the preferred parameter settings are as follows: the diagonal element of the process noise covariance matrix of the extended Kalman filter is 0.02, the diagonal element of the measurement noise covariance matrix of the extended Kalman filter is 0.1, the preset frame rate is 70%, and the feature fusion frequency is 10.

[0146] The localization results can be evaluated using absolute trajectory error (ATE) and relative pose error (RPE). ATE calculates the difference in translational components of the pose at each timestamp by directly aligning the estimated trajectory with the true trajectory in the global coordinate system. This difference is the root mean square error (RMSE) of the three-dimensional Euclidean distance between all pose points after alignment, reflecting the cumulative localization deviation of the algorithm in the global coordinate system. RPE focuses on the local consistency of pose changes between adjacent points within a fixed time interval, calculating the difference in relative pose transformation between the estimated and true trajectories within the same time interval to quantify the degree of system drift. RPE includes translational and rotational components, with the RMSE of the translational component typically used as the primary metric.

[0147] Based on the above-described multimodal visual positioning method, this application also provides a multimodal visual positioning device. The following will combine... Figure 4 The device is described in detail.

[0148] Figure 4 A structural block diagram of a multimodal visual positioning device according to an embodiment of this application is shown.

[0149] like Figure 4 As shown, the multimodal visual positioning device 400 of this embodiment includes a feature extraction module 410, a feature prediction module 420, a feature noise reduction module 430, a feature fusion module 440, and a device positioning module 450.

[0150] The feature extraction module 410 is used to extract features from the image data acquired by the image acquisition device mounted on the vehicle at a target time in response to the acquisition of image data, thereby obtaining image features. The image data includes visual images, and the image features include visual features. In one embodiment, the feature extraction module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0151] The feature prediction module 420 is used to predict the depth features at the target time based on the expected pose transformation of the image acquisition device and historical depth features, thereby obtaining predicted depth features. Historical depth features are obtained by feature extraction from historical depth images, which are depth images acquired by the image acquisition device at historical times. The expected pose transformation is predicted based on the historical pose transformation of the image acquisition device. In one embodiment, the feature prediction module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0152] The feature denoising module 430, when it is determined that the image data also includes a depth image, performs denoising processing on the predicted depth features based on the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features. The depth features are obtained by feature extraction from the depth image. In one embodiment, the feature denoising module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0153] The feature fusion module 440 is used to fuse optimized deep features and visual features to obtain multimodal visual features. In one embodiment, the feature fusion module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0154] The device positioning module 450 is used to locate the image acquisition device based on multimodal visual features to obtain the positioning result at the target time. In one embodiment, the device positioning module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0155] According to an embodiment of this application, the feature prediction module 420 includes a trajectory determination submodule and a feature prediction submodule.

[0156] The trajectory determination submodule is used to determine the motion trajectory of the image acquisition device from the historical moment to the target moment based on the desired pose change.

[0157] The feature prediction submodule is used to predict the depth features of the target time based on historical depth features and motion trajectory, and obtain the predicted depth features.

[0158] According to an embodiment of this application, the multimodal visual positioning device 400 further includes a matrix determination module.

[0159] The matrix determination module is used to determine the covariance matrix that characterizes the confidence of the predicted depth features based on the state transition matrix, the historical covariance matrix, and the noise covariance matrix that characterize the motion of the image acquisition device. The historical covariance matrix is ​​the covariance matrix at a historical time.

[0160] According to an embodiment of this application, the feature denoising module 430 includes a matrix determination submodule, a gain determination submodule, and a feature denoising submodule.

[0161] The matrix determination submodule is used to determine the measurement matrix based on the mapping relationship. The measurement matrix is ​​used to describe the degree of influence of changes in depth features on the predicted depth features.

[0162] The gain determination submodule is used to determine the Kalman gain based on the measurement matrix and the covariance matrix.

[0163] The feature denoising submodule is used to denoise the predicted depth features based on the Kalman gain and measurement noise to obtain optimized depth features. The measurement noise is the residual between the depth features and the predicted depth features.

[0164] According to an embodiment of this application, the multimodal visual positioning device 400 further includes a visual fusion module.

[0165] The visual fusion module is used to fuse predicted depth features and visual features to obtain multimodal visual features when the image data does not include depth images.

[0166] According to embodiments of this application, the multimodal visual positioning device 400 further includes a feature matching module, a feature point determination module, a parameter determination module, and a pose determination module.

[0167] The feature matching module is used to perform feature matching between multimodal visual features and historical multimodal visual features to determine the correspondence between feature points in the multimodal visual features and historical multimodal visual features. Historical multimodal visual features represent multimodal visual features at historical moments.

[0168] The feature point determination module is used to determine a preset number of feature point groups from multimodal visual features and historical multimodal visual features based on the correspondence relationship. Each feature point group includes a feature point from the multimodal visual features and a historical feature point from the historical multimodal visual features that corresponds to the feature point.

[0169] The parameter determination module is used to determine the rotation matrix and translation vector of the image acquisition device based on the two-dimensional coordinates of the feature points, the three-dimensional coordinates of the historical feature points, and the intrinsic parameters of the image acquisition device.

[0170] The pose determination module is used to determine the pose transformation of the image acquisition device based on the rotation matrix and translation vector.

[0171] According to an embodiment of this application, the device positioning module 450 includes a device positioning submodule.

[0172] The device positioning submodule is used to determine the positioning result at the target time based on the pose transformation and the historical positioning results of the image acquisition device at historical moments.

[0173] According to embodiments of this application, any multiple modules among the feature extraction module 410, feature prediction module 420, feature noise reduction module 430, feature fusion module 440, and device positioning module 450 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the feature extraction module 410, feature prediction module 420, feature noise reduction module 430, feature fusion module 440, and device positioning module 450 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the feature extraction module 410, feature prediction module 420, feature denoising module 430, feature fusion module 440, and device positioning module 450 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0174] Figure 5 A block diagram of an electronic device suitable for implementing a multimodal visual positioning method according to an embodiment of this application is shown.

[0175] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0176] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0177] According to embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0178] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0179] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0180] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0181] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0182] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0183] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0184] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0186] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0187] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A multimodal visual localization method, characterized in that, The method includes: In response to the image acquisition device mounted on the vehicle acquiring image data at a target time, feature extraction is performed on the image data to obtain image features, wherein the image data includes visual images and the image features include visual features; Based on the expected pose transformation of the image acquisition device and the historical depth features, the depth features at the target time are predicted to obtain the predicted depth features. The historical depth features are obtained by feature extraction from the historical depth image, which is a depth image acquired by the image acquisition device at a historical time. The expected pose transformation is predicted based on the historical pose transformation of the image acquisition device. If it is determined that the image data also includes the depth image, the predicted depth features are denoised according to the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features, wherein the depth features are obtained by feature extraction from the depth image; The optimized depth features and the visual features are fused to obtain multimodal visual features; and Based on the multimodal visual features, the image acquisition device is located to obtain the location result of the target time.

2. The method according to claim 1, characterized in that, Based on the expected pose transformation of the image acquisition device and historical depth features, the depth features at the target time are predicted to obtain the predicted depth features, including: Based on the desired pose transformation, determine the motion trajectory of the image acquisition device from the historical moment to the target moment; and Based on the historical depth features and the motion trajectory, the depth features at the target time are predicted to obtain the predicted depth features.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the state transition matrix, historical covariance matrix, and noise covariance matrix that characterize the motion of the image acquisition device, a covariance matrix for characterizing the confidence of the predicted depth feature is determined, wherein the historical covariance matrix is ​​the covariance matrix at the historical moment.

4. The method according to claim 3, characterized in that, The step of performing noise reduction processing on the predicted depth features based on the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features includes: Based on the mapping relationship, a measurement matrix is ​​determined, which is used to describe the degree of influence of the changes in the depth features on the predicted depth features; Based on the measurement matrix and the covariance matrix, determine the Kalman gain; and The predicted depth features are denoised based on the Kalman gain and measurement noise to obtain the optimized depth features, where the measurement noise is the residual between the depth features and the predicted depth features.

5. The method according to claim 1, characterized in that, The method further includes: If it is determined that the image data does not include the depth image, the predicted depth features and the visual features are fused to obtain the multimodal visual features.

6. The method according to claim 1, characterized in that, The method further includes: The multimodal visual features and historical multimodal visual features are matched to determine the correspondence between feature points in the multimodal visual features and historical multimodal visual features. The historical multimodal visual features represent the multimodal visual features at the historical moment. Based on the correspondence, a preset number of feature point groups are determined from the multimodal visual features and the historical multimodal visual features. Each feature point group includes a feature point in the multimodal visual features and a historical feature point in the historical multimodal visual features that corresponds to the feature point. Based on the two-dimensional coordinates of the feature points, the three-dimensional coordinates of the historical feature points, and the intrinsic parameters of the image acquisition device, determine the rotation matrix and translation vector of the image acquisition device; and Based on the rotation matrix and the translation vector, the pose transformation of the image acquisition device is determined.

7. The method according to claim 6, characterized in that, The step of locating the image acquisition device based on the multimodal visual features to obtain the location result of the target time includes: Based on the pose transformation and the historical positioning results of the image acquisition device at the historical time, the positioning result at the target time is determined.

8. The method according to claim 1, characterized in that, The feature fusion method for fusing the optimized depth features and the visual features includes at least one of the following: feature concatenation, feature weighted summation.

9. A multimodal visual positioning device, characterized in that, The device includes: The feature extraction module is used to extract features from the image data acquired by the image acquisition device mounted on the vehicle at a target time in response to the image data being acquired by the image acquisition device on the vehicle at a target time, and to obtain image features, wherein the image data includes visual images and the image features include visual features. The feature prediction module is used to predict the depth features at the target time based on the expected pose change of the image acquisition device and the historical depth features, so as to obtain the predicted depth features. The historical depth features are obtained by feature extraction from the historical depth image, which is a depth image acquired by the image acquisition device at a historical time. The expected pose change is predicted based on the historical pose change of the image acquisition device. The feature denoising module is used to, when it is determined that the image data also includes the depth image, perform denoising processing on the predicted depth features according to the mapping relationship between the depth features and the predicted depth features to obtain optimized depth features, wherein the depth features are obtained by feature extraction from the depth image; The feature fusion module is used to fuse the optimized deep features and the visual features to obtain multimodal visual features; and The device positioning module is used to locate the image acquisition device based on the multimodal visual features and obtain the positioning result at the target time.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images