Cross-domain autonomous positioning method and device for air-ground amphibious robot based on frustum transformation

Abstract

The application discloses a kind of based on frustum transformation air-ground amphibious robot cross-domain autonomous positioning method and device, it is related to intelligent robot autonomous navigation and multimodal environment perception technical field, the method includes: obtaining multilevel feature vector database;Real-time acquisition air-ground amphibious robot current height, current attitude angle and current visual image;Projection transformation matrix is constructed, perspective transformation is carried out, and geometric correction image is obtained;Extract visual feature vector, text semantic feature vector and environmental text information and its confidence;The matching result of each level is calculated respectively;The dynamic fusion weight of ground layer, transition layer and high altitude layer is calculated;Interest point information and prompt word in matching result are input into visual-language big model, and generate thought chain decision log;Based on dynamic fusion weight, weighted fusion is carried out, and the final positioning coordinate is output.The application realizes high-precision cross-domain autonomous positioning by dynamic frustum correction, hierarchical semantic alignment and explainable inference.

Description

Technical Field

[0001] This application relates to the field of autonomous navigation and multimodal environmental perception technology for intelligent robots, and in particular to a cross-domain autonomous localization method and device for amphibious robots based on view cone transformation. Background Technology

[0002] With the booming development of the low-altitude economy, amphibious robots, as a new type of intelligent carrier combining the characteristics of ground mobile robots and rotary-wing drones, are gradually being widely used in logistics distribution, urban security patrols, and disaster relief. These robots possess unique cross-domain mobility capabilities, meaning that when performing tasks, they need to switch between ground driving mode and low-altitude flight mode, and undergo a continuous vertical movement process from takeoff and landing at cruising altitude. High-precision self-localization is a prerequisite for robots to complete their tasks. Traditional positioning technology mainly relies on the Global Navigation Satellite System (GNSS). However, in complex urban environments, GNSS signals are easily blocked and interfered with, leading to positioning failures. Therefore, visual geolocation based on airborne cameras has become the mainstream alternative or auxiliary solution.

[0003] However, existing visual positioning technologies still have significant shortcomings when applied to vertical cross-domain scenarios for amphibious robots. Firstly, during vertical ascent and descent, the onboard camera's perspective abruptly changes from a ground-level view to an aerial view, resulting in severe perspective distortion. Traditional matching algorithms, optimized only for a single fixed viewpoint, struggle to handle such dramatic changes, easily leading to inaccurate visual feature matching. Secondly, semantic information at different altitudes cannot be effectively integrated. Existing technologies largely rely excessively on basic image texture features; valuable information such as ground text and road signs becomes ineffective after the robot takes off due to blurring or perspective shift, lacking a mechanism to transfer ground semantic information to the air. Furthermore, existing georeferenced databases primarily consist of two independent, discrete data sets: "ground street view images" and "high-altitude satellite images." They lack three-dimensional image texture reference data for low-altitude robot transitions, resulting in a lack of corresponding reference data for positioning matching during the intermediate stages of ascent or descent, creating positioning blind spots. Meanwhile, traditional positioning systems mostly use fixed weights when fusing multi-source positioning results, and cannot dynamically adjust the level of trust according to actual conditions such as flight altitude and environmental clarity. In addition, conventional positioning algorithms have "black box" characteristics, making it difficult to trace the cause of anomalies when positioning deviations occur.

[0004] Therefore, there is an urgent need for a method that can adapt to the cross-scene perception needs of low-altitude robots, dynamically adjust the geometric projection of the view cone during vertical movement, and combine the thinking chain to realize the hierarchical correspondence of semantic information at different heights, thereby achieving high-precision autonomous positioning. Summary of the Invention

[0005] The purpose of this application is to provide a cross-domain autonomous localization method and device for amphibious robots based on view cone transformation, which solves the problems of viewpoint distortion, semantic discontinuity, localization blind zone and uninterpretable results in vertical motion. Through dynamic view cone correction, hierarchical semantic alignment and interpretable reasoning, high-precision cross-domain autonomous localization is achieved.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] In a first aspect, this application provides a cross-domain autonomous localization method for an air-ground amphibious robot based on view cone transformation, comprising: acquiring an offline constructed multi-level feature vector database, wherein the multi-level feature vector database includes feature vectors corresponding to the ground layer, transition layer, and high-altitude layer respectively; acquiring the current height, current attitude angle, and current visual image of the air-ground amphibious robot in real time; performing perspective transformation on the current visual image using a projection transformation matrix based on the current height and the current attitude angle to obtain a geometrically corrected image; inputting the geometrically corrected image into a vision-language large model, and extracting visual feature vectors, text semantic feature vectors, and environmental text information and their confidence levels in parallel; calculating the visual feature vectors and the transition layer feature vectors, the current attitude angle, and the current visual image of the air-ground amphibious robot respectively; and calculating the current height, current attitude angle, and current visual image of the air-ground amphibious robot based on a projection transformation matrix to obtain a geometrically corrected image; inputting the geometrically corrected image into a vision-language large model, and extracting visual feature vectors, text semantic feature vectors, and environmental text information and their confidence levels in parallel; and calculating the current height, current attitude angle, and current visual image of the air-ground amphibious robot respectively. The similarity of the empty layer feature vectors and the similarity between the text semantic feature vector and the ground layer feature vector are calculated to obtain the matching results for each layer. Based on the current height, the confidence level of the environmental text information, and the matching results for each layer, the dynamic fusion weights of the ground layer, transition layer, and high-altitude layer are calculated. Data containing at least the current height, the environmental text information, the interest point information in the ground layer matching results, and the dynamic fusion weights are encapsulated into prompt words and input into the visual-language large model to generate a thought chain decision log for verifying the matching results and the rationality of the weight allocation. Based on the dynamic fusion weights, the positioning coordinates corresponding to the matching results of each layer are weighted and fused to output the final positioning coordinates of the amphibious robot.

[0008] Optionally, the constructed multi-level feature vector database includes: acquiring vector road network data, 3D model data, and satellite image data of the target area; generating a natural language description based on the metadata of points of interest in the vector road network data, and extracting the feature vector of the natural language description through the text encoder of the visual-language large model as the ground layer feature vector; rendering a sequence of tilted-view images at a preset altitude and pitch angle based on the 3D model data, and extracting the feature vector of the tilted-view image sequence through the visual encoder of the visual-language large model as the transition layer feature vector; performing grid tiling based on the satellite image data, and extracting the feature vector of each tile image through the visual encoder of the visual-language large model as the upper-air layer feature vector.

[0009] Optionally, the three-dimensional model data is a city-level high-precision triangular mesh model.

[0010] Optionally, determining the target pitch angle of the virtual camera based on the current height includes: establishing a non-linear mapping relationship between the current height and the target pitch angle using a Sigmoid function; when the current height is the ground height, the target pitch angle is a horizontal angle; when the current height exceeds a preset height threshold, the target pitch angle is a vertical angle.

[0011] Optionally, the extraction of environmental text information and its confidence level includes: using the language decoding head of the vision-language large model to generate a text sequence based on the geometrically corrected image; extracting the generation probability of each word in the text sequence, and calculating the geometric mean of the generation probability of each word, and using the geometric mean as the confidence level of the environmental text information.

[0012] Optionally, the calculation of the dynamic fusion weights of the ground layer, transition layer, and upper-air layer includes: calculating the comprehensive evaluation values ​​of the ground layer, transition layer, and upper-air layer respectively based on the current altitude, the confidence level of the environmental text information, and the matching results of each layer, wherein the comprehensive evaluation value of the ground layer is positively correlated with the confidence level of the environmental text information and negatively correlated with the current altitude; the comprehensive evaluation value of the transition layer reaches its peak when the current altitude is at a preset optimal observation altitude; the comprehensive evaluation value of the upper-air layer is positively correlated with the current altitude; and the dynamic fusion weights are obtained by normalizing the comprehensive evaluation values ​​of the ground layer, transition layer, and upper-air layer using the Softmax function.

[0013] Optionally, the generation of the thought chain decision log includes a reasoning process consisting of four stages: visual cross-validation, description consistency verification, weight rationality verification, and final positioning decision.

[0014] Optionally, the current attitude angle includes roll angle, pitch angle, and yaw angle.

[0015] Optionally, the similarity is calculated by a normalized dot product operation.

[0016] Secondly, this application provides a cross-domain autonomous localization device for an amphibious robot based on frustum transformation, comprising: a database storage module for storing an offline-constructed multi-level feature vector database, the multi-level feature vector database including feature vectors corresponding to the ground layer, transition layer, and high-altitude layer respectively; a perception data acquisition module for acquiring the current height, current attitude angle, and current visual image of the amphibious robot in real time; a geometric correction module for performing perspective transformation on the current visual image using a projection transformation matrix based on the current height and current attitude angle to obtain a geometrically corrected image; a multimodal feature extraction module loaded with a visual-language large model for inputting the geometrically corrected image into the visual-language large model and extracting visual feature vectors, text semantic feature vectors, and environmental text information and their confidence scores in parallel; and a similarity calculation module for calculating the visual feature vectors, text semantic feature vectors, and environmental text information and their confidence scores respectively. The similarity between the feature vector and the feature vector of the transition layer and the feature vector of the upper-level layer, and the similarity between the text semantic feature vector and the feature vector of the ground layer are calculated to obtain the matching results of each level; the dynamic weight calculation module is used to calculate the dynamic fusion weight of the ground layer, the transition layer and the upper-level layer based on the current height, the confidence of the environmental text information and the matching results of each level; the decision reasoning module is used to encapsulate data containing at least the current height, the environmental text information, the interest point information in the ground layer matching results and the dynamic fusion weight into prompt words, input them into the visual-language big model, and generate a thought chain decision log for verifying the rationality of the matching results and weight allocation; the positioning fusion output module is used to perform weighted fusion of the positioning coordinates corresponding to the matching results of each level based on the dynamic fusion weight, and output the final positioning coordinates of the amphibious robot.

[0017] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0018] This application provides a cross-domain autonomous localization method and device for amphibious robots based on frustum transformation. By determining the target pitch angle of the virtual camera according to the current altitude and constructing a projection transformation matrix to perform perspective transformation on the current visual image, it can eliminate the image perspective distortion caused by drastic changes in perspective during the vertical take-off and landing of the amphibious robot in real time, and compensate for the geometric deviation between the ground-level view and the high-altitude view, thereby significantly improving the success rate of image feature matching under complex perspectives and providing high-quality image input for subsequent localization. Secondly, by constructing a multi-level feature vector database containing ground layer, transition layer and high-altitude layer, and combining visual feature vectors, text semantic feature vectors and environmental text information extracted in parallel by a large vision-language model, it realizes the effective connection of semantic features at different altitude levels. This architecture solves the feature failure problem that easily occurs at a specific altitude due to a single data source, and ensures the continuity and stability of the robot's localization in the entire space from the ground to the low altitude. Furthermore, by calculating the feature similarity at each level and dynamically fusion weights based on the current altitude, the confidence level of environmental text information, and the matching results at each level, the reliability of the localization results at each level can be adaptively adjusted according to the robot's current flight altitude and the quality of environmental information. This effectively overcomes the localization jump problem easily caused by traditional hard threshold switching methods, achieving a smooth transition and stable, robust fusion of multi-source localization results. Finally, by encapsulating multi-source data into prompt words and inputting them into a visual-language large model, a thought chain decision log is generated to verify the rationality of the matching results and weight allocation. This achieves dual verification of numerical calculation and semantic logic, solving the black-box uninterpretability problem of traditional localization systems. It provides an intuitive and clear basis for fault backtracking when localization anomalies occur, significantly improving the credibility and reliability of the entire localization system. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an application environment diagram of a cross-domain autonomous localization method for an air-ground amphibious robot based on view cone transformation, according to one embodiment of this application.

[0021] Figure 2 A flowchart illustrating a cross-domain autonomous localization method for an air-ground amphibious robot based on view cone transformation, provided as an embodiment of this application;

[0022] Figure 3 A flowchart illustrating the construction process of a multi-level feature vector database provided in an embodiment of this application;

[0023] Figure 4 Target pitch angle of a virtual camera provided in an embodiment of this application With real-time altitude A schematic diagram of the mapping curve;

[0024] Figure 5 This is a schematic diagram of a virtual viewpoint projection transformation process provided in an embodiment of this application;

[0025] Figure 6 This is a structural diagram of a cross-modal feature extraction and dynamic weight calculation module provided in an embodiment of this application;

[0026] Figure 7 A flowchart illustrating the offline hierarchical feature database construction and online cross-domain autonomous localization reasoning process provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the functional modules of an air-ground amphibious robot cross-domain autonomous positioning device based on view cone transformation, provided as an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] This application provides a cross-domain autonomous localization method for air-ground amphibious robots based on view cone transformation, which can be applied to, for example... Figure 1In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on another server. Terminal 101 can send the real-time acquired current height, current attitude angle, and current visual image of the amphibious robot to server 102 via the network. After receiving the real-time perception data, server 102 performs dynamic frustum geometry correction on the received current visual image based on an offline-constructed multi-level feature vector database, extracts cross-modal semantic features, calculates hierarchical feature similarity, determines dynamic fusion weights, and performs decision reasoning based on a thought chain to finally obtain the robot's positioning coordinates. Server 102 can then feed back the obtained final positioning coordinates to terminal 101. Furthermore, in some embodiments, the above positioning method can also be implemented independently by server 102 or terminal 101. For example, terminal 101 can directly execute all positioning steps on the real-time perception data to be processed, or server 102 can obtain the pre-constructed multi-level feature vector database from the data storage system and then perform positioning processing.

[0031] The terminal 101 can be, but is not limited to, various desktop computers, laptops, smartphones, smartwatches, etc. The server 102 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0032] In one exemplary embodiment, such as Figure 2 As shown, a cross-domain autonomous localization method for an air-ground amphibious robot based on view cone transformation is provided. This method is executed by a computer device and includes the following steps:

[0033] S1, obtain the offline constructed multi-level feature vector database, which includes feature vectors corresponding to the ground layer, transition layer and upper-level layer respectively.

[0034] S2 acquires the current height, current attitude angle, and current visual image of the amphibious robot in real time.

[0035] S3. Based on the current height and the current attitude angle, a perspective transformation is performed on the current visual image using a projection transformation matrix to obtain a geometrically corrected image. Specifically, based on the current height, the target pitch angle of the virtual camera is determined, and a projection transformation matrix is ​​constructed based on the current attitude angle and the target pitch angle. The projection transformation matrix is ​​then used to perform a perspective transformation on the current visual image to obtain a geometrically corrected image.

[0036] S4 inputs the geometrically corrected image into the visual-language large model and extracts visual feature vectors, text semantic feature vectors, and environmental text information and their confidence levels in parallel.

[0037] S5 calculates the similarity between the visual feature vector and the transition layer feature vector, the upper layer feature vector, and the text semantic feature vector and the ground layer feature vector, respectively, to obtain the matching results for each level.

[0038] S6 calculates the dynamic fusion weights of the ground layer, transition layer, and upper-air layer based on the current altitude, the confidence level of environmental text information, and the matching results of each layer.

[0039] S7 encapsulates data containing at least the current altitude, environmental text information, interest point information from the ground layer matching results, and dynamic fusion weights into prompt words, inputs them into the visual-language big model, and generates a thought chain decision log to verify the matching results and the rationality of the weight allocation.

[0040] S8, based on dynamic fusion weights, performs weighted fusion of the positioning coordinates corresponding to the matching results at each level, and outputs the final positioning coordinates of the amphibious robot.

[0041] By implementing steps S1 to S8, and through step S3 where the virtual camera target pitch angle is determined based on the current altitude and a projection transformation matrix is ​​constructed to perform perspective transformation on the current visual image, the perspective distortion of the image caused by drastic changes in viewpoint during the vertical take-off and landing of the amphibious robot can be eliminated in real time. This compensates for the geometric deviation between the ground-level viewpoint and the high-altitude viewpoint, thereby significantly improving the success rate of image feature matching under complex viewpoints and providing high-quality image input for subsequent positioning. Secondly, by constructing a multi-level feature vector database containing ground, transition, and high-altitude layers through step S1, and combining it with the visual feature vectors, text semantic feature vectors, and environmental text information extracted in parallel by the visual-language large model in step S4, effective connection of semantic features at different altitude levels is achieved. This architecture solves the feature failure problem that easily occurs with a single data source at a specific altitude, ensuring the continuity and stability of the robot's positioning in the entire spatial domain from the ground to low altitude. Furthermore, by calculating the feature similarity at each level in step S5, and by calculating the dynamic fusion weights in step S6 based on the current altitude, the confidence level of environmental text information, and the matching results at each level, the reliability of the localization results at each level can be adaptively adjusted according to the robot's current flight altitude and the quality of environmental information. This mechanism effectively overcomes the localization jump problem easily caused by the traditional hard threshold switching method, achieving a smooth transition and stable and robust fusion of multi-source localization results. Finally, by encapsulating the multi-source data into a visual-language large model with prompt words in step S7, a thought chain decision log is generated to verify the rationality of the matching results and weight allocation, achieving dual verification of numerical calculation and semantic logic. This method solves the black-box uninterpretability problem of traditional localization systems, providing an intuitive and clear basis for fault backtracking when localization anomalies occur, and significantly improving the credibility and reliability of the entire localization system.

[0042] In practice, step S1 employs an offline preprocessing method, constructing a feature vector database with three levels based on the publicly available geographic information dataset CityNav. ,in, Represented as the feature vector of the ground layer, This is represented as the feature vector of the transition layer. This is represented as a feature vector of the upper atmosphere.

[0043] In practical implementation, the specific construction process of the multi-level feature vector database refers to... Figure 3 As shown, it includes:

[0044] S11: Acquire vector road network data, 3D model data, and satellite imagery data of the target area.

[0045] Understandably, the CityNav simulation environment uses three types of basic data obtained through Python scripts and standard API interfaces. Specifically, vector road network data is obtained by calling the Overpass API to acquire OSM vector data within the target area, which is then parsed to obtain data containing nodes. id The system includes a list of nodes for lat, lon, and tags; a 3D model data loading system for a high-precision triangular mesh model of the CityNav dataset, which includes building geometry and surface texture; and satellite imagery data from CityNav ZoomLevel 20 satellite orthophotos with a resolution better than 0.3 meters per pixel.

[0046] S12 generates a natural language description based on the metadata of interest points in the vector road network data, and extracts the feature vector of the natural language description as the ground layer feature vector through the text encoder of the vision-language big model.

[0047] Understandably, by extracting point-of-interest nodes with labels such as name, facility type, and store from OSM vector data, the extracted metadata is input into the Qwen2.5-VL large model to generate a natural language description T. desc The generated long text description T desc The text encoder input to the Qwen2.5-VL model is mapped to ground layer feature vectors. .in, In the formula, This is the text encoder for the Qwen2.5-VL model, where R represents the set of real numbers and D represents the dimension of the vector. It is represented as a D-dimensional real vector space.

[0048] S13, based on 3D model data, renders and generates a sequence of tilted view images at a preset height and pitch angle, and extracts the feature vector of the tilted view image sequence through the visual encoder of the visual-language large model, which serves as the feature vector of the transition layer.

[0049] Understandably, by using 3D scan models from the CityNav dataset as the data source, constructing a rendering pipeline using the PyTorch3D open-source library, setting virtual camera parameters (height 20 meters, pitch angle -45°), and performing batch projection rendering along the road network trajectory, a sequence of two-dimensional tilted view images covering the target area is generated. obl The rendered image sequence is then input into the visual encoder of the Qwen2.5-VL model to extract transition layer feature vectors containing texture information of the building's side profile. .in, In the formula, The visual encoder is represented as the Qwen2.5-VL model.

[0050] S14, based on satellite imagery data, performs grid slicing, and extracts the feature vectors of each slice image through the visual encoder of the vision-language large model, which are used as the upper-level feature vectors.

[0051] Understandably, by using CityNav's Zoom Level 20 satellite orthophotos as the data source, and employing a 512m × 512m sliding window with a 50% overlap rate for grid tiling, the tiled images are preprocessed (resized to 336 × 336 to fit the model input). The preprocessed images are then input into the Qwen2.5-VL model visual encoder to obtain the upper-level feature vectors. .in, .

[0052] In specific implementation, the geometrically corrected image obtained in step S13 is specifically as follows: based on the real-time height Calculate the target pitch angle of the virtual camera .like Figure 4 The target pitch angle of the virtual camera shown With real-time altitude The schematic diagram of the mapping curve uses the Sigmoid function for smooth transition, and the calculation formula is as follows:

[0053] ;

[0054] In the above formula, k is the view switching rate coefficient, and in this embodiment, k=0.2; In this embodiment, the intermediate height threshold is used for viewpoint transition. =15 meters. when When ≈0 (ground level), ≈0 (eye level); when >30 meters ≈-π / 2 (vertical top view).

[0055] Then, the camera coordinate system is set according to standard computer vision conventions (X-axis horizontal to the right, Y-axis vertical downwards, Z-axis forward along the optical axis), such as... Figure 5 The virtual viewpoint projection transformation process shown uses the camera intrinsic parameter K and the difference in the rotation matrix to construct the projection transformation matrix. The formula for calculating this matrix is:

[0056] ;

[0057] In the above formula, K is the camera intrinsic parameter matrix; The rotation matrix representing the current true attitude obtained by the airborne IMU; Let be the rotation matrix of the target's virtual viewpoint, and its mathematical expression is:

[0058] ;

[0059] Finally, using For the original image Perform perspective transformation to eliminate projection distortion caused by vertical motion, and obtain a geometrically corrected image. Specifically, for the target image Each pixel coordinate (x, y) in the image is obtained using... Find it in the source image floating-point coordinates Pixel values ​​are calculated using bilinear interpolation. The calculation formula is:

[0060] ;

[0061] In the above formula, For distance-weighted bilinear interpolation coefficients, i and j are index variables used to iterate through the four neighboring source pixels around the target pixel.

[0062] In specific implementation, the feature extraction in step S14 is as follows: In the visual feature extraction branch, the image corrected in step S3 is... Inputting Qwen-2.5VL, the model performs dynamic resolution alignment on the image using the following formula: H and W represent the height and width of the original input image. and These represent the height and width of the aligned image, respectively. This indicates a rounding up operation. The adjusted pixel tensor generates an initial feature sequence through the image tile embedding module (PatchEmbed). Initial feature sequence The calculation formula is: ,in, This represents the weight matrix of the pre-trained convolutional layer. This is the bias vector. After deep feature extraction through 32 Transformer hidden layers, the data stream enters the feature merging module (PatchMerger) to perform a linear transformation, resulting in dimensionality-reduced features. The linear transformation formula is: ,in, Representing spatial coordinates Local feature vectors at; This indicates a concatenation operation along the channel dimension; This represents the projection weight matrix of the feature aggregation layer. Finally, a 1024-dimensional real-time visual feature vector is output through a global attention pooling layer. The calculation formula is: In the formula, Indicates global average pooling. This represents the dimension reduction projection matrix.

[0063] In the text feature processing branch, the real-time input natural language instructions will be processed. Input the text processing branch of Qwen2.5-VL. Utilize the tokenizer to perform a byte-pair encoding algorithm, mapping the instruction text into a discrete sequence of token indices. The input is a text encoder based on the Transformer architecture. Semantic fusion is performed to extract the hidden state at the position of the sequence end marker, and then normalized projection is performed. The calculation formula is as follows: In the formula, This represents the context hidden vector output by the text encoder at the end-of-words position; The alignment projection matrix representing the text modality; Norm( )express Normalization operation; This represents the final output text semantic feature vector.

[0064] In the environmental text information OCR extraction branch, the Qwen2.5-VL language decoding head is used to generate a sequence of visible text descriptions in the current scene using a greedy search strategy, denoted as a string. To quantify the reliability of textual landmarks in the environment, the original logarithmic probability of each word during generation is extracted and transformed into conditional probability using the Softmax function. Let the generated word sequence be... The confidence index is calculated using the following formula: In the formula, The model represents the first time. The predicted character token generated step by step; Indicates the total length of the generated text sequence; This represents the Softmax conditional probability output by the model. This represents the OCR confidence scalar calculated based on the geometric mean; if the image contains road sign text with clear outlines and explicit meaning... The closer the value is to 1, the faster the score will decay.

[0065] In specific implementation, the process of calculating similarity in step S5 is as follows: for all vectors L2 normalization is performed, and the normalization calculation formula is as follows: In the formula, This represents the input feature vector to be normalized. Representing vectors of Norm (Euclidean norm); This is a tiny constant used to prevent the denominator from being zero; This represents the normalized output vector, whose magnitude is always 1. Output the normalized vector: .

[0066] Then, matrix multiplication is used to simultaneously calculate the matching score vectors at three levels: ground semantic matching degree. ; Transition layer matching degree High-altitude matching degree .

[0067] Finally, based on the calculated three matching degree vectors, the maximum value and corresponding index of the similarity scores at the three levels are extracted, and then used from the pre-built feature library. Latitude and longitude corresponding to the backtracking matching item: ground layer Transition layer Upper atmosphere .

[0068] In specific implementation, the process of calculating the dynamic fusion weights of the ground layer, transition layer, and upper-level layer in step S6 is as follows: combining real-time altitude... OCR confidence The feature matching degree of the three levels is used to calculate the comprehensive evaluation value of the ground layer, transition layer, and upper layer respectively:

[0069] ;

[0070] ;

[0071] ;

[0072] in, This is a comprehensive evaluation value for the ground layer. This is the comprehensive evaluation value for the transition layer. This is a comprehensive assessment value for the upper atmosphere; This is the text gain coefficient, in this embodiment... 10; For high attenuation coefficient, =0.5; To match the confidence coefficient, =5; Based on the basic bias constant, =5; The optimal observation height for the transition layer, =20 meters; For the corresponding height bandwidth parameters, =100; The peak bias constant is =8; For the height gain coefficient, =0.3; As the activation threshold, =10.

[0073] Then, the unbounded overall evaluation value is mapped to a probability weight vector with a sum of 1 using the Softmax function. The calculation formula is as follows: In the formula, These represent the dynamic fusion weights for the ground layer, transition layer, and upper atmosphere layer, respectively. The structure diagram of the cross-modal feature extraction and dynamic weight calculation module described above can be seen as follows: Figure 6 As shown.

[0074] In specific implementation, the reasoning process of the thought chain decision log in step S7 is as follows: Constructing prompt words ,and ,in, The name of the point of interest that best matches the ground layer. The corresponding natural language description will then be provided. Then, the prompt words will be... Input the Qwen2.5-VL model to generate an inference log that includes four stages: visual cross-validation, description consistency verification, weight rationality verification, and final localization decision.

[0075] In specific implementation, the process of reasoning the final coordinates in step S8 is as follows: Based on dynamic fusion weights, the latitude and longitude coordinates of the best matching items in each feature layer are weighted and fused to output the robot's final absolute positioning latitude and longitude coordinates. The calculation formula is as follows:

[0076] ;

[0077] in, , The latitude and longitude of the viewpoint corresponding to the transition layer 3D model; , The latitude and longitude of the center point of the upper-level satellite image slice are used to calculate the final fused latitude and longitude. , Furthermore, the aforementioned offline hierarchical feature database construction and online cross-domain autonomous localization inference process can be performed as follows: Figure 7 As shown.

[0078] To verify the effectiveness of this application, a quantitative evaluation was conducted on the CityNav simulation dataset, outputting four categories of metrics: recall R@1 (first-choice hit rate), recall R@5 (top-five hit rate), precision, and average localization error (APE). Comparative experiments with existing methods NetVLAD, CVM-Net, L2LTR, and AnyLoc show that the proposed method achieves an R@1 value of 90.82%, a 4.32 percentage point improvement over the second-best method AnyLoc, and reduces the average localization error (APE) to 2.85 meters, demonstrating the superiority of this application in vertical spatial cross-domain scenarios.

[0079] Ablation experiments further validated the contributions of each module. Using the scheme using only Qwen2.5-VL as the baseline (R@1=87.15%, APE=4.50m), after sequentially adding the dynamic frustum geometry correction module, semantic hierarchical alignment module, and CoT decision and dynamic fusion module, R@1 increased to 88.40%, 89.65%, and 90.82%, respectively, and APE decreased to 4.25m, 3.80m, and 2.85m, respectively, proving that each module in this application has a positive effect on improving positioning accuracy.

[0080] In one exemplary embodiment, such as Figure 8 As shown, a cross-domain autonomous localization device for an air-ground amphibious robot based on view cone transformation is provided, comprising:

[0081] The database storage module 201 is used to store a multi-level feature vector database constructed offline, the multi-level feature vector database including feature vectors corresponding to the ground layer, transition layer and upper-level layer respectively.

[0082] The perception data acquisition module 202 is used to acquire the current height, current attitude angle and current visual image of the amphibious robot in real time.

[0083] The geometric correction module 203 is used to perform perspective transformation on the current visual image based on the current height and the current attitude angle using a projection transformation matrix to obtain a geometrically corrected image. Specifically, it determines the target pitch angle of the virtual camera based on the current height, constructs a projection transformation matrix based on the current attitude angle and the target pitch angle, and uses the projection transformation matrix to perform perspective transformation on the current visual image to obtain a geometrically corrected image.

[0084] The multimodal feature extraction module 204 is loaded with a large visual-language model, which is used to input the geometrically corrected image into the large visual-language model and extract visual feature vectors, text semantic feature vectors, and environmental text information and their confidence levels in parallel.

[0085] The similarity calculation module 205 is used to calculate the similarity between the visual feature vector and the transition layer feature vector and the upper-level feature vector, and to calculate the similarity between the text semantic feature vector and the ground layer feature vector, so as to obtain the matching results of each level.

[0086] The dynamic weight calculation module 206 is used to calculate the dynamic fusion weights of the ground layer, transition layer and high-altitude layer based on the current altitude, the confidence level of the environmental text information and the matching results of each layer.

[0087] The decision reasoning module 207 is used to encapsulate data containing at least the current height, the environmental text information, the interest point information in the ground layer matching result, and the dynamic fusion weight into prompt words, input them into the visual-language big model, and generate a thought chain decision log for verifying the rationality of the matching result and weight allocation.

[0088] The positioning fusion output module 208 is used to perform weighted fusion of the positioning coordinates corresponding to the matching results at each level based on the dynamic fusion weight, and output the final positioning coordinates of the air-ground amphibious robot.

[0089] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a cross-domain autonomous localization method for amphibious robots based on frustum transformation.

[0090] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0091] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0092] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0093] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Furthermore, any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A cross-domain autonomous localization method for an air-ground amphibious robot based on view cone transformation, characterized in that, include: Obtain an offline-constructed multi-level feature vector database, which includes feature vectors corresponding to the ground layer, transition layer, and upper-level layer, respectively. The constructed multi-level feature vector database includes: acquiring vector road network data, 3D model data, and satellite imagery data of the target area; generating natural language descriptions based on metadata of points of interest in the vector road network data, and extracting feature vectors of the natural language descriptions using a text encoder of a visual-language large model as the ground layer feature vectors; rendering and generating a sequence of tilted-view images at a preset altitude and pitch angle based on the 3D model data, and extracting feature vectors of the tilted-view image sequences using a visual encoder of the visual-language large model as the transition layer feature vectors; performing grid tiling based on the satellite imagery data, and extracting feature vectors of each tile image using a visual encoder of the visual-language large model as the upper-air layer feature vectors; Real-time acquisition of the amphibious robot's current altitude, current attitude angle, and current visual image; Based on the current height and the current attitude angle, a perspective transformation is performed on the current visual image using a projection transformation matrix to obtain a geometrically corrected image; The geometrically corrected image is input into the visual-language large model, and visual feature vectors, text semantic feature vectors, environmental text information and their confidence scores are extracted in parallel. The similarity between the visual feature vector and the transition layer feature vector and the upper layer feature vector is calculated respectively, and the similarity between the text semantic feature vector and the ground layer feature vector is calculated to obtain the matching results of each layer. Based on the current altitude, the confidence level of the environmental text information, and the matching results of each level, the dynamic fusion weights of the ground layer, transition layer, and upper-air layer are calculated. This includes: calculating the comprehensive evaluation values ​​of the ground layer, transition layer, and upper-air layer respectively based on the current altitude, the confidence level of the environmental text information, and the matching results of each level; wherein the comprehensive evaluation value of the ground layer is positively correlated with the confidence level of the environmental text information and negatively correlated with the current altitude; the comprehensive evaluation value of the transition layer reaches its peak when the current altitude is at a preset optimal observation altitude; the comprehensive evaluation value of the upper-air layer is positively correlated with the current altitude; and the dynamic fusion weights are obtained by normalizing the comprehensive evaluation values ​​of the ground layer, transition layer, and upper-air layer using the Softmax function. The data, which includes at least the current height, the environmental text information, the interest point information in the ground layer matching results, and the dynamic fusion weights, is encapsulated into prompt words and input into the visual-language big model to generate a thought chain decision log for verifying the matching results and the rationality of the weight allocation. Based on the dynamic fusion weights, the positioning coordinates corresponding to the matching results at each level are weighted and fused to output the final positioning coordinates of the amphibious robot.

2. The cross-domain autonomous localization method for amphibious robots based on view cone transformation according to claim 1, characterized in that, The three-dimensional model data is a city-level high-precision triangular mesh model.

3. The cross-domain autonomous localization method for amphibious robots based on view cone transformation according to claim 1, characterized in that, Determining the target pitch angle of the virtual camera based on the current altitude includes: A nonlinear mapping relationship between the current altitude and the target pitch angle is established using the Sigmoid function. When the current altitude is the ground altitude, the target pitch angle is a horizontal angle. When the current height exceeds a preset height threshold, the target pitch angle is a vertical pitch angle.

4. The cross-domain autonomous localization method for amphibious robots based on view cone transformation according to claim 1, characterized in that, Extract environmental text information and its confidence level, including: Using the language decoding head of the aforementioned vision-language large model, a text sequence is generated based on the geometrically corrected image; Extract the generation probability of each word in the text sequence, calculate the geometric mean of the generation probability of each word, and use the geometric mean as the confidence level of the environmental text information.

5. The cross-domain autonomous localization method for amphibious robots based on view cone transformation according to claim 1, characterized in that, The generation of the thought chain decision log includes a reasoning process consisting of four stages: visual cross-validation, description consistency verification, weight rationality verification, and final positioning decision.

6. The cross-domain autonomous localization method for amphibious robots based on view cone transformation according to claim 1, characterized in that, The current attitude angles include roll angle, pitch angle, and yaw angle.

7. The cross-domain autonomous localization method for amphibious robots based on view cone transformation according to claim 1, characterized in that, The similarity is calculated using a normalized dot product operation.

8. A cross-domain autonomous localization device for an amphibious robot based on view cone transformation, characterized in that, The cross-domain autonomous localization device for amphibious robots based on frustum transformation refers to the cross-domain autonomous localization method for amphibious robots based on frustum transformation according to any one of claims 1-7, and the cross-domain autonomous localization device for amphibious robots based on frustum transformation includes: The database storage module is used to store a multi-level feature vector database constructed offline, which includes feature vectors corresponding to the ground layer, transition layer and upper-level layer respectively. The perception data acquisition module is used to acquire the current height, current attitude angle, and current visual image of the amphibious robot in real time. The geometric correction module is used to perform perspective transformation on the current visual image based on the current height and the current attitude angle using a projection transformation matrix to obtain a geometrically corrected image; The multimodal feature extraction module is loaded with a large visual-language model, which is used to input the geometrically corrected image into the large visual-language model and extract visual feature vectors, text semantic feature vectors, and environmental text information and their confidence levels in parallel. The similarity calculation module is used to calculate the similarity between the visual feature vector and the transition layer feature vector and the upper layer feature vector, respectively, and to calculate the similarity between the text semantic feature vector and the ground layer feature vector, so as to obtain the matching results of each level. The dynamic weight calculation module is used to calculate the dynamic fusion weights of the ground layer, transition layer and high-altitude layer based on the current altitude, the confidence level of the environmental text information and the matching results of each layer. The decision reasoning module is used to encapsulate data containing at least the current height, the environmental text information, the interest point information in the ground layer matching results, and the dynamic fusion weights into prompt words, input them into the visual-language big model, and generate a thought chain decision log for verifying the matching results and the rationality of the weight allocation; The positioning fusion output module is used to perform weighted fusion of the positioning coordinates corresponding to the matching results at each level based on the dynamic fusion weight, and output the final positioning coordinates of the air-ground amphibious robot.

Images