Real-time three-dimensional visual positioning method and system based on multi-modal fusion

Abstract

The application discloses a kind of real-time three-dimensional visual positioning method and system based on multimodal fusion, it is related to computer vision and multimodal perception technical field, including: obtaining the two-dimensional image stream containing target object and the natural language query instruction describing target object;Extract the visual feature of two-dimensional image stream, extract the text feature of natural language query instruction;Based on visual feature, predict the depth probability distribution of each pixel;Based on depth probability distribution, the visual feature of two-dimensional image stream is promoted to three-dimensional space, and three-dimensional voxel feature is generated by aggregation;Three-dimensional voxel feature is processed, and target voxel feature is obtained;Text feature and target voxel feature are fused in cross-modal feature, and target object is positioned in three-dimensional space and output positioning result.The application reduces the dependence on pre-acquisition point cloud data, realizes lightweight and low-delay real-time streaming three-dimensional perception closed loop through feature processing and cross-modal fusion of multimodal feature.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal perception technology, and in particular to a real-time three-dimensional visual localization method and system based on multimodal fusion. Background Technology

[0002] 3D visual localization is used to establish a mapping relationship between natural language descriptions and 3D physical space coordinates. This technology requires perception to not only extract visual representations of 2D images, but also to possess cross-modal fine-grained semantic alignment capabilities and 3D spatial geometric reconstruction capabilities. In scenarios such as autonomous localization of mobile robots, indoor visual navigation, and augmented reality, achieving low-latency spatial target retrieval based on real-time single-view or multi-view visual streams is the perceptual foundation for machines to understand complex open scenes.

[0003] Currently, research on 3D visual localization mainly falls into two major architectural categories. One is a two-stage matching architecture based on point cloud input, which relies on LiDAR or depth sensors to pre-scan and acquire complete scene point cloud data, generating global 3D candidate boxes and performing cross-modal association matching with text features. The other is an end-to-end single-stage architecture based on image centers, which directly extracts features from the 2D image stream, uses a pure vision-based depth prediction network to estimate the depth probability distribution of each pixel, projects the 2D features along the camera ray into 3D space through an outer product operation, aggregates them into a dense voxel mesh, and performs cross-modal attention computation on the full voxel space to achieve target object localization.

[0004] Current technologies suffer from a disconnect between deep inference and language querying, leading to haphazard feature construction and enormous computational overhead. Therefore, further research and innovation are needed to address these issues in existing technologies. Summary of the Invention

[0005] Purpose of the invention: To provide a real-time 3D visual positioning method and system based on multimodal fusion, in order to solve the above-mentioned problems in the prior art.

[0006] Technical solution: Firstly, a real-time 3D visual localization method based on multimodal fusion, comprising:

[0007] Acquire a 2D image stream containing the target object, and a natural language query command describing the target object;

[0008] Extract visual features from 2D image streams and extract textual features from natural language query commands;

[0009] Based on visual features, predict the depth probability distribution of each pixel;

[0010] Based on the depth probability distribution, visual features are enhanced to three-dimensional space and aggregated to generate three-dimensional voxel features;

[0011] The three-dimensional voxel features are processed to obtain the target voxel features;

[0012] Cross-modal feature fusion of text features and target voxel features is performed to locate the target object in three-dimensional space and output the localization result.

[0013] In conjunction with the first aspect, a real-time 3D vision positioning system based on multimodal fusion includes:

[0014] At least one processor;

[0015] Memory that is communicatively connected to at least one processor;

[0016] The memory stores instructions that can be executed by a processor to implement the method described in any of the first aspects.

[0017] Beneficial effects: This invention reduces the reliance on pre-acquired point cloud data and achieves a lightweight and low-latency real-time streaming 3D perception closed loop through feature processing of multimodal features and cross-modal fusion. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a real-time 3D visual positioning method based on multimodal fusion, provided for an embodiment of this application.

[0019] Figure 2 The flowchart provided in this application illustrates how visual features are upscaled to three-dimensional space and aggregated to generate three-dimensional voxel features based on depth probability distribution.

[0020] Figure 3 This is a flowchart illustrating the prediction of the depth probability distribution of each pixel based on visual features, as provided in an embodiment of this application.

[0021] Figure 4 This is a flowchart illustrating the process of processing three-dimensional voxel features to obtain target voxel features, as provided in an embodiment of this application.

[0022] Figure 5 This is a flowchart illustrating how text features and target voxel features are fused across modalities to locate a target object in three-dimensional space and output the localization result, as provided in this embodiment of the application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a predetermined order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, product, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] To address the aforementioned issues, the applicant conducted in-depth searches and analyses, and discovered:

[0026] Existing methods for depth distribution prediction rely on purely visual features, uniformly constructing a 3D feature volume across the entire field of view. A large number of irrelevant background voxels serve as noise candidates in subsequent complex cross-modal calculations. Specifically, the mechanism of blindly constructing global 3D features first and then locally matching language results in the feature energy of the target object in 3D space being diluted by incorrectly projected depth noise. This easily leads to ambiguity in monocular observations lacking depth sensors, resulting in a low signal-to-noise ratio for the target region features.

[0027] Furthermore, performing cross-modal attention fusion computation directly on a fully dense 3D voxel mesh increases computational complexity cubically with spatial resolution, resulting in significant memory consumption and computational redundancy. This directly hinders the engineering deployment of low-latency real-time streaming inference on computing-constrained terminals.

[0028] To solve these problems, combined with Figures 1 to 5 The present invention will be specifically described through the following embodiments.

[0029] On the one hand, an exemplary scheme for a real-time 3D visual localization method based on multimodal fusion is provided, specifically including:

[0030] Step 101: Obtain a two-dimensional image stream containing the target object, and a natural language query command describing the target object.

[0031] In this embodiment, the process of acquiring a two-dimensional image stream containing the target object is implemented using the hardware acquisition module of the mobile terminal. Specifically, the AR glasses worn by the user or the smartphone in their hand activates its built-in red-green-blue three-channel color RGB camera to continuously capture environmental images within the field of view at a preset frame rate, forming a two-dimensional image stream. The frame rate can be optionally set to 30 frames per second. Simultaneously, the mobile terminal's microphone captures the user's voice commands, which are then converted into text format by the built-in speech recognition engine to obtain a natural language query command describing the target object.

[0032] For example, after acquiring a two-dimensional image stream, the mobile terminal uses a hardware video encoder, such as an H.265 standard encoder, to compress it in real time. The compressed video stream and natural language query instructions are encapsulated into a synchronous data packet and sent to a cloud server for further processing via a mobile communication network, such as a 5G network, using a low-latency streaming media transmission protocol such as WebRTC.

[0033] Alternatively, real-time compression can also be performed using common video coding standards.

[0034] Step 102: Extract visual features from the two-dimensional image stream and extract text features from the natural language query instructions;

[0035] After receiving the decoded data, the cloud server extracts features from both the visual and textual modalities. For the two-dimensional image stream, it invokes a pre-deployed backbone convolutional neural network, such as the ResNet series of residual networks, to process the image data frame by frame. Multi-layer convolution and pooling operations are used to extract edge, texture, and semantic abstract information contained in the image, outputting a multi-scale two-dimensional feature tensor, i.e., visual features.

[0036] For example, this embodiment provides two optional processing paths for text feature extraction of natural language query instructions. Optionally, a direct encoding path can be used. When the input instruction structure is simple and the direction is clear, such as the instruction for a white coffee cup, a pre-built lightweight text encoder, such as the text branch of the contrastive language-image pre-trained CLIP model, can be used to perform word segmentation and embedding operations on the natural language query instruction, and output a fixed-dimensional feature vector as text features.

[0037] In other words, text features are obtained by directly encoding the natural language query instructions using a pre-built text encoder.

[0038] Correspondingly, a disambiguation path based on a large language model can also be adopted. When a user input instruction is detected to have ambiguous referents or complex spatial logical relationships, such as the container next to the blue backpack where one can drink water, a pre-built large language model (LLM) can be used to perform logical inference and disambiguation on the instruction. Based on its built-in commonsense reasoning capabilities, the large language model parses the description and converts it into structured query text, such as the water cup next to the blue backpack.

[0039] Next, the structured query text is input into a text encoder for feature encoding, resulting in high-dimensional text features. This cascaded text processing mechanism ensures fast response while also improving the ability to understand open vocabulary and complex semantics.

[0040] Step 103: Based on visual features, predict the depth probability distribution of each pixel;

[0041] In other words, predict the depth probability distribution of each pixel.

[0042] After extracting visual features, depth information needs to be assigned to each pixel in the 2D image to complete the transition from 2D to 3D. Accordingly, the visual features are input into the depth prediction module. This module discretizes the continuous physical space into multiple depth bins along a preset depth direction. For any pixel on the feature map, the depth prediction module calculates the probability value of that pixel falling within each depth bin. The combination of probability values ​​from all depth bins constitutes the depth probability distribution of that pixel. This distribution reflects the inference of the pixel's distance from the camera in 3D space.

[0043] Step 104: Based on the depth probability distribution, the visual features are uplifted to three-dimensional space and aggregated to generate three-dimensional voxel features; this step may include:

[0044] Obtain the camera's intrinsic parameter matrix and extrinsic pose matrix in real time;

[0045] The visual features are multiplied by the depth probability distribution to generate three-dimensional point features at each discrete depth.

[0046] By combining the camera's intrinsic and extrinsic pose matrices, the 3D point features are projected onto a unified world coordinate system to obtain the projected 3D point features.

[0047] The projected 3D point features falling within the same spatial grid are aggregated to generate 3D voxel features.

[0048] Furthermore, it is necessary to obtain the camera's intrinsic parameter matrix and extrinsic pose matrix, which are acquired in real time by the mobile terminal. The camera's intrinsic parameter matrix includes optical parameters such as the camera's focal length and principal point coordinates; the extrinsic pose matrix is ​​calculated in real time by the mobile terminal's built-in inertial measurement unit (IMU) or visual inertial odometry (VIO), reflecting the camera's rotation and translation state in the world coordinate system at the moment of image acquisition.

[0049] Based on this, the extracted visual features are multiplied by the predicted depth probability distribution. This operation uses the depth probability as weights to scatter the two-dimensional features of a pixel onto its corresponding camera ray, generating three-dimensional point features at each discrete depth.

[0050] Next, combining the camera's intrinsic and extrinsic pose matrices, a rigid body transformation is performed on the 3D point features, projecting them from the camera coordinate system to a unified world coordinate system. After spatial projection, a 3D spatial mesh is predefined in the world coordinate system. The 3D coordinates of the geometric center of each spatial mesh in the world coordinate system are the 3D physical anchor point coordinates corresponding to that voxel.

[0051] By iterating through all the 3D point features, all 3D point features falling within the same spatial grid, i.e., within a voxel, are summed or subjected to other forms of pooling aggregation operations to generate the initial 3D voxel features.

[0052] Furthermore, a pre-built 3D convolutional neural network can be used to model the local spatial relationships of the initial 3D voxel features. The 3D convolutional kernel slides in the voxel grid, capturing the geometric continuity and semantic correlation between adjacent voxels, and outputs refined 3D voxel features, which are then used as the input basis for subsequent processing steps.

[0053] Step 105: Process the three-dimensional voxel features to obtain the target voxel features;

[0054] In other words, feature selection or pass-through is performed based on three-dimensional voxel features to obtain target voxel features;

[0055] In this embodiment, the aggregated 3D voxel features undergo pass-through processing, meaning that the 3D voxel features containing full environmental information are directly used as target voxel features for subsequent localization calculations. This processing method preserves the integrity of the scene and is suitable for cloud environments with sufficient computing power or localization tasks with relatively simple scenes.

[0056] Step 106: Perform cross-modal feature fusion between text features and target voxel features to locate the target object in three-dimensional space and output the localization result.

[0057] Textual features representing query intent and target voxel features representing the 3D scene are fed into a cross-modal localization network. In this network, the features from the two modalities are deeply fused through an attention mechanism or matrix multiplication. The network then searches for voxel locations with strong responses by calculating the matching degree between text semantics and various regions in 3D space.

[0058] Furthermore, the cross-modal localization network regresses the center coordinates, length, width, height, and yaw angle of the target object's 3D bounding box in 3D space, forming a localization result. This result is then transmitted back to the mobile terminal via a low-latency downlink. Upon receiving the result, the mobile terminal utilizes an AR rendering engine, such as a mobile augmented reality core engine, to overlay the 3D bounding box onto the user's preview screen in real time, completing the virtual-real fusion localization feedback loop.

[0059] According to one aspect of this application, a real-time 3D visual positioning system based on multimodal fusion includes:

[0060] At least one processor;

[0061] Memory that is communicatively connected to at least one processor;

[0062] The memory stores instructions that can be executed by a processor to implement the method of the present invention.

[0063] On the other hand, a language-guided deep prediction mechanism based on Bayesian posterior fusion is described. This mechanism solves the deep ambiguity problem in traditional pure visual inference by fusing global language priors with pixel-level visual features. This can include:

[0064] Step 201: Map the visual features using a pre-built deep prediction network to obtain the visual basic distribution;

[0065] Accordingly, visual features are received. These visual features are specifically represented as a multi-channel three-dimensional tensor format. A pre-built depth prediction network is used to extract local feature vectors for each pixel location on this feature map. Specifically, the depth prediction network can be implemented using a fully convolutional structure with a 1×1 kernel size.

[0066] As an example, this network linearly maps the channel dimensions of visual features to a predefined set of discrete depth bins. After processing with a normalized exponential function, it outputs the depth probability prediction results under purely visual conditions, thus obtaining the visual fundamental distribution. This distribution represents the probability of each pixel falling within different depth distance intervals, inferred solely from the image's appearance and texture features.

[0067] In some implementations, a spatial attention mechanism can be cascaded within the depth prediction network to smooth the visual baseline distribution by leveraging the local geometric coherence of adjacent pixels, thereby reducing depth inference fluctuations caused by local image occlusion or weak texture regions.

[0068] Step 202: Process the text features using a pre-built deep prior network to obtain the global linguistic conditional prior distribution;

[0069] In other words, while processing visual information, the user-input query data is processed in parallel. A pre-built deep prior network is used to receive text features, specifically high-dimensional feature vectors that aggregate global semantics. The deep prior network contains a multi-layer perceptron structure that maps high-dimensional semantic space features to the same discrete depth box space as the visual prediction.

[0070] Furthermore, exponential normalization is used to calculate the global linguistic conditional prior distribution. This distribution is a globally shared one-dimensional probability vector that quantifies the implicit physical depth range constraints in a text query. For example, when the text query is "a water glass on a table," this distribution vector will exhibit a higher probability response at nearby depth bin indices.

[0071] Furthermore, the mapping matrix parameters of the deep prior network are obtained through pre-training iterations using large-scale scene graphs and depth-annotated data, thereby recording the pre-existing distance prior statistical patterns of predetermined items in physical space.

[0072] Step 203: Calculate the pixel-level correlation gating value for each pixel based on the cross-modal bilinear interaction of visual and text features;

[0073] This process requires evaluating the relevance of each local region in the image to the query text to avoid erroneous spatial guidance from prior linguistic information on the image background. Accordingly, feature interaction computation is performed at the individual pixel level. For any pixel in the feature map, its corresponding visual feature vector is extracted and subjected to cross-modal bilinear interaction computation with the text features.

[0074] Optionally, this interactive operation employs a learnable bilinear projection matrix to map the features of the two modalities into a unified metric space for dot product matching. To prevent excessively large dot product values ​​from causing gradient vanishing during subsequent backpropagation, a scaling factor can be introduced to normalize the dot product results.

[0075] Based on this, the normalized interaction results are input into the activation function for non-linear mapping, compressing their values ​​to the effective range of 0 to 1, and calculating the pixel-level correlation gating value corresponding to each pixel, which reflects the degree of semantic matching between the visual texture of the predetermined pixel and the text query intent.

[0076] Specifically, when a pixel falls within the target object region, the calculated gate value approaches 1; when a pixel belongs to an irrelevant background region, the gate value approaches 0.

[0077] As an example, the calculation of pixel-level correlation gating values ​​can be described by the following formula:

[0078] g _u,v =σ((f _u,v ) T ×W _g ×f _q’ / sqrt(C _v ));

[0079] Among them, g _u,v To calculate the pixel-level correlation gating value of the output, σ is the Sigmoid activation function, (f _u,v ) T W is the transpose of the visual feature vectors. _g f is a learnable bilinear interaction matrix used for spatial mapping. _q’ C represents the text feature vector after feature alignment. _v The number of channels for visual features.

[0080] Step 204: Based on pixel-level correlation gating values, the visual basic distribution and the linguistic conditional prior distribution are fused to obtain the depth probability distribution.

[0081] Furthermore, Bayesian posterior fusion is performed on the three sets of data sources. The visual baseline distribution is used as the baseline likelihood term, and the linguistic conditional prior distribution is used as the conditional prior term. For each pixel in the two-dimensional plane, its corresponding pixel-level correlation gating value is extracted as an exponential modulation weight. In the logarithmic probability space, this gating value is used to perform multiplicative weighted modulation on the prior term, and then added to the likelihood term. The summation result is then mapped back to the linear probability space through a normalized exponential function, outputting a normalized depth probability distribution. The specific calculation is implemented using the following posterior fusion formula:

[0082] α lang _u,v =Softmax(logα vis _u,v +g _u,v ×logp prior );

[0083] Where, α lang _u,v Let α be the depth probability distribution of pixels after incorporating language information, Softmax be the normalized exponential function, log be the natural logarithm function, and α be the depth probability distribution of pixels. vis _u,vFor a purely visual fundamental distribution, g _u,v p is the pixel-level correlation gating value calculated in the preceding step. prior Let be the language conditional prior distribution vector.

[0084] Taking the target object pixel as an example, when there is depth ambiguity in pure visual inference and the probability quality is dispersed in two depth intervals, if the semantics of the text query strongly points to a certain predetermined depth range, the calculated pixel-level relevance gate value will approach 1. The posterior fusion formula will superimpose the linguistic condition prior distribution as a high-weight modulation term onto the logarithmic space of the visual basic distribution. After normalization exponential function mapping, the probability quality of the depth interval corresponding to the target semantics is concentrated, and the depth inference ambiguity is eliminated.

[0085] Conversely, when processing pixels in the background region of an image, due to semantic mismatch, the pixel-level correlation gating value approaches 0, and the posterior fusion result automatically degenerates into the visual basic distribution. The linguistic prior does not interfere with the depth inference of irrelevant regions.

[0086] In other embodiments, when processing pixels in the image background region, the calculated pixel-level correlation gating value will approach 0 due to semantic mismatch. In this case, the prior terms in the fusion formula have lower weights, and the posterior fusion result will automatically degenerate and be strictly equivalent to the visual baseline distribution. This mechanism ensures that the linguistic prior will not produce erroneous depth guidance for the spatial geometry construction of irrelevant regions within the visual field.

[0087] This invention introduces a language-guided depth prediction mechanism based on Bayesian posterior fusion. This technique, in the process of upscaling 2D visual features to 3D space, fuses the global linguistic conditional prior distribution with pixel-level visual features through a matching gating value. It departs from the existing approach of blindly constructing 3D features across the entire scene, actively focusing the depth probability distribution onto the actual physical depth range of the query target. This can suppress spatial depth ambiguity under monocular observation and improve the signal-to-noise ratio of the 3D features of the target region.

[0088] On another front, the adaptive voxel sparsity mechanism for query-aware processing is described. This mechanism evaluates the cross-modal semantic matching degree between the 3D feature space and the text query, dynamically adjusts the filtering threshold using information entropy, and removes voxel data from irrelevant regions, thereby reducing the computational resource consumption of the subsequent cross-modal localization network. Specific steps include:

[0089] Step 301: Directly use the three-dimensional voxel features of the full state as the target voxel features;

[0090] Alternatively, the semantic correlation between 3D voxel features and text features can be calculated, and adaptive voxel sparsification can be performed based on this (semantic correlation between 3D voxel features and text features) to obtain the target voxel features in a sparse state.

[0091] In this embodiment, after spatial aggregation is completed, a branch path for feature processing is selected based on the current computing resource status or configuration mode. In one processing branch, the full-state 3D voxel features are directly passed to the downstream network as target voxel features. This branch is suitable for cloud processing environments with sufficient computing power.

[0092] In another processing branch, in order to reduce the computational load of real-time processing on mobile devices, the semantic correlation between 3D voxel features and text features is calculated, and adaptive voxel sparsification screening is performed accordingly to obtain the target voxel features in a sparse state.

[0093] This filtering mechanism can filter out background voxels that are irrelevant to the user's query intent in advance, and compress the physical retrieval space for subsequent positioning from a global grid to a set of highly relevant local grids.

[0094] Step 302: Calculate the voxel-level semantic relevance score for each voxel based on the cross-modal bilinear interaction between 3D voxel features and text features.

[0095] Accordingly, for any voxel within the three-dimensional spatial grid, its corresponding multi-dimensional feature vector is extracted, and it is used to perform cross-modal bilinear interaction computation with the text features that aggregate global semantics. This interaction process utilizes a learnable feature interaction matrix to project the three-dimensional visual representation and the one-dimensional text representation into the same metric dimension space, and calculates their inner product to obtain the original matching response value.

[0096] Next, using a non-linear activation function, such as the Sigmoid activation function, the original response value is mapped to a probability range of 0 to 1, and the voxel-level semantic relevance score corresponding to each voxel is calculated and output.

[0097] Step 303: Perform three-dimensional spatial coherence smoothing on the voxel-level semantic relevance score to obtain the smoothed relevance score.

[0098] In the physical world, target objects typically occupy a continuous three-dimensional spatial volume, and their feature correlation distribution should possess the coherent properties of physical space. If threshold screening is performed directly based on independently calculated single-element scores, isolated interference noise points are easily retained due to local feature calculation noise.

[0099] Based on this, a three-dimensional convolutional layer is used to perform three-dimensional spatial coherence smoothing on the voxel-level semantic relevance score. A three-dimensional smoothing kernel of size (3,3,3) is constructed, and a weighted average of the score values ​​of adjacent voxels is calculated in the three spatial dimensions of length, width, and height, thereby obtaining the smoothed relevance score.

[0100] It should be understood that this smoothing process suppresses isolated discrete background noise, resulting in a uniform and enhanced response for connected element sets corresponding to the same physical target region.

[0101] Step 304: Calculate the information entropy of the smoothed correlation score according to the formula H=-1 / N×∑[r×log2(r)+(1-r)×log(1-r)], where H is the information entropy, N is the total number of all elements, r is the smoothed correlation score of each voxel, and log2 is the logarithm to the base 2.

[0102] Furthermore, an objective index representing the matching uncertainty is calculated based on the distribution characteristics of the global scoring data. The entire set of voxels is traversed, and the numerical values ​​are substituted into the information entropy formula to perform a summation operation. The binary entropy value of each voxel reflects the degree of uncertainty in the correlation score of that voxel; H∈[0,1], H=0 indicates a polarized score distribution, and H=1 indicates the maximum uncertainty state where all voxel scores are 0.5.

[0103] It should be understood that this metric quantifies the degree of dispersion in the alignment between current scene features and instruction semantics. When there are multiple interfering objects in the scene that are similar to the query target, or when the target features are blurred due to occlusion, the distribution of scoring data tends to be uniform, and the calculated information entropy value is relatively large; when the target object features are large and unique, the distribution of scoring data is polarized, and the calculated information entropy value is relatively small.

[0104] Step 305, according to the formula K=K _base +β×H adaptively calculates the current voxel retention ratio, where K is the current voxel retention ratio. _base β is the pre-configured base retention ratio, and β is the pre-configured adjustable coefficient; wherein, the retention scale of subsequent sparsification processing is dynamically determined based on the acquired information entropy using a linear control mechanism.

[0105] Another example: Suppose that in a certain positioning cycle, the information entropy H of the smoothed relevance score distribution is 0.8. Read the pre-configured base retention ratio K. _base The set value is 0.05, and the pre-configured adjustable coefficient β is set to 0.1. Substituting the parameters into the retention ratio formula, we can calculate K = 0.05 + 0.1 × 0.8 = 0.13.

[0106] The calculation results show that, due to the high uncertainty in the current scene matching, the current voxel retention ratio is adaptively increased from the baseline of 0.05 to 0.13, retaining more edge voxels to prevent missed detections. In contrast, if the information entropy H value calculated for another scene is 0.2, the retention ratio calculated after substituting it into the formula decreases to 0.07, and a higher intensity voxel pruning calculation is performed accordingly.

[0107] Step 306: Obtain the descending distribution of the smoothed correlation score among all voxels, and take the quantile corresponding to the current voxel retention ratio as the confidence threshold; set the binary filter mask corresponding to the voxels whose smoothed correlation score is greater than or equal to the confidence threshold to the valid state.

[0108] All voxels within the spatial grid are sorted globally in descending order based on their smoothed correlation scores. The quantiles at the corresponding percentage positions are used as a comparison benchmark, based on the current voxel retention ratio from the previous output. For example, if the current voxel retention ratio is 0.13, the voxel scores at the top 13% critical positions are extracted as the confidence threshold.

[0109] Based on this, the relationship between the overall voxel score and the threshold is compared one by one. For voxels whose smoothed correlation score is greater than or equal to the confidence threshold, their corresponding binary filter mask is set to the valid state, specifically represented by a value of 1; for voxels whose score is lower than the threshold, their corresponding binary filter mask is set to the invalid state, specifically represented by a value of 0.

[0110] Step 307: Perform hard selection filtering on the three-dimensional voxel features based on the binary filtering mask, perform soft weighting of features using the smoothed correlation score, and refine the features using three-dimensional sparse convolution operation to obtain the sparse target voxel features.

[0111] After obtaining the state mask of each voxel, joint culling and enhancement processing is performed on the 3D voxel features. Hard selection filtering is performed using a binary filtering mask to release the voxel data blocks corresponding to invalid states from the memory computation graph, cutting off the data flow path of background voxels to participate in downstream attention computation.

[0112] For voxels whose states are valid and preserved, their corresponding smoothed relevance scores are used as multiplication coefficients to perform soft-weighted feature calculations on their feature vectors. This weighting calculation results in voxels with higher semantic matching degrees receiving stronger feature magnitude representations.

[0113] Furthermore, for the selected and retained subset of non-zero voxel data, local feature refinement is performed using 3D sparse convolution. 3D sparse convolution only performs dot product operations within non-zero voxels and their local effective neighborhoods, avoiding redundant computation on idle 3D grids and outputting sparse target voxel features.

[0114] In this embodiment, a query-aware adaptive voxel sparsity mechanism is employed. A cross-modal bilinear interactive computation is used to calculate the relevance score between the 3D space and the text semantics. The information entropy of the score distribution is used to adaptively and dynamically adjust the voxel retention threshold, pre-eliminating a large number of irrelevant background voxels before they enter the attention network. This mechanism compresses the global physical retrieval space into a highly relevant local sparse grid, mitigating the problem of computational complexity increasing sharply with spatial resolution and achieving low-latency streaming 3D visual localization.

[0115] In some embodiments, a possible implementation method of a network architecture based on cross-modal bidirectional attention localization and voxel-side regression is described. This method introduces a three-dimensional sinusoidal position encoding and residual connection mechanism to directly regress local coordinate offsets at the voxel physical anchor point, thereby achieving three-dimensional spatial localization of the target object. Exemplarily, this embodiment can be implemented by performing the following steps:

[0116] Step 401: Using three-dimensional sinusoidal position coding based on world coordinates, spatial position information is injected into the target voxel features to generate a position-aware three-dimensional representation.

[0117] Optionally, the target voxel features obtained from the previous processing are acquired. Due to the sparsification or convolutional aggregation processing, the data structure of these features mainly represents semantic and local geometric information, lacking absolute spatial physical positioning information. Therefore, the three-dimensional physical anchor point coordinates of each target voxel in the real-world coordinate system are extracted.

[0118] Optionally, a three-dimensional sinusoidal position coding function is used to perform high-dimensional mapping calculations on the coordinates of the three-dimensional physical anchor point. For each dimensional component of the three-dimensional physical anchor point coordinates, the value of the component is encoded and mapped using sine and cosine functions of different frequencies.

[0119] It should be understood that the encoding results of each dimension are concatenated to generate a spatial location embedding vector with the same feature dimensions as the target voxel.

[0120] Next, the spatial location embedding vector is added element-wise to the target voxel features to generate a position-aware 3D representation. This process allows the downstream network to simultaneously perceive the absolute position reference of the voxel in real physical space while processing semantic information.

[0121] Step 402: Using the location-aware 3D representation as the query and the text features as the key and value, perform the first stage of cross-modal attention calculation to obtain the language-enhanced voxel representation.

[0122] The location-aware 3D representation is used as the query matrix input for the attention mechanism, and text features are used as the key and value matrices input. A first learnable projection matrix and a second learnable projection matrix are further constructed. The first learnable projection matrix is ​​used to linearly map the query matrix, and the second learnable projection matrix is ​​used to linearly map the key matrix, thus unifying the voxel feature dimension and text feature dimension into a predefined shared attention dimension space.

[0123] In the same metric space, scaled dot product attention computation is performed to generate cross-modal association mappings between text lexical units and target voxels.

[0124] Step 403: Extract the corresponding attention output features based on cross-modal attention calculation;

[0125] In this step, the obtained mapping results are used to perform a weighted summation of the value matrix. Specifically, the semantic features of each text word are weighted and combined according to their importance to the current voxel, and the corresponding attention output features are output. The attention output features represent the contextual semantic information extracted and absorbed by the current physical voxel from the global language instructions.

[0126] Step 404: The attention output features and the position-aware 3D representation are summed by residual connection, and the summed features are used as the voxel representation for language enhancement, preserving the original spatial geometric information of the 3D voxel features.

[0127] If attention output features are used directly to perform subsequent parameter prediction calculations, the original three-dimensional spatial coordinate resolution capability of voxels will be masked by global language information.

[0128] Based on this, a residual connection topology is adopted to perform joint summation calculations on the calculated attention output features and the original position-aware 3D representation. This summation process is specifically implemented through the following formula:

[0129] Z _1 =Z+A _1 ×(F _Q ×W _V1 );

[0130] Among them, Z _1 Z is the language-enhanced voxel representation matrix for the output, Z is the position-aware 3D representation matrix for the input, and A is the output voxel representation matrix for the output. _1 To calculate the attention weight matrix output for cross-modal dot product, F _Q Given the input text feature matrix, W _V1 Let be the third learnable projection matrix used to map text features to the value space.

[0131] Alternatively, Z is the position-aware 3D representation matrix projected onto the shared attention dimension space via the first learnable projection matrix, and its dimension is the same as A. _1 ×(F _Q ×W _V1 The output dimensions are consistent, ensuring that the residual summation operation is performed within a unified feature dimension space. Conventional dimension alignment projection methods, such as linear mapping layers, can be used to achieve dimension unification.

[0132] It should also be understood that, through the residual computation architecture, the data representation of each voxel, while integrating the target language matching information, is physically forced to retain the initial three-dimensional mesh coordinate anchor point data, thus avoiding the degradation of the feature space dimension.

[0133] Step 405: For each language-enhanced voxel representation, use a pre-built target scoring network to predict the target score, and use a pre-built bounding box regression network to predict the 3D bounding box offset and size.

[0134] Specifically, on the voxel feature side that maintains the 3D physical structure, direct regression calculation of localization parameters is performed. For each language-enhanced voxel representation existing within the mesh, it is synchronously transferred to two independent multilayer perceptron computation branches.

[0135] As an example, the first computational branch constitutes a pre-built target scoring network, which calculates through multi-level linear mappings and outputs a probability assessment value of whether the current voxel belongs to the target object, i.e., the target score.

[0136] As another example, the second computational branch constitutes a pre-built bounding box regression network that predicts 3D physical geometry parameters in parallel for the current voxel. Specific output parameters include the spatial offset of the target center point relative to the coordinates of the current voxel's 3D physical anchor point, as well as scalar dimension data for the target's length, width, and height.

[0137] Step 406: Obtain the coordinates of the three-dimensional physical anchor points of each voxel in the world coordinate system;

[0138] The voxel with the highest target score is selected, and its corresponding 3D bounding box offset and size, along with the 3D physical anchor point coordinates, are used to calculate and generate the positioning result.

[0139] The above steps involve performing a descending comparison of the target scores calculated for all voxels, extracting the voxel with the highest score. The 3D bounding box offset and dimensions corresponding to this voxel, as well as the 3D physical anchor point coordinates of this voxel in the spatial mesh, are obtained. The 3D bounding box offset is used as a relative displacement component, and a 3D vector addition operation is performed with the 3D physical anchor point coordinates to calculate the absolute center coordinates of the target object in the world coordinate system. The calculation of these absolute center coordinates can be expressed as:

[0140] (c _x ,c _y ,c _z )=(x _world ,y _world ,z _world )+(Δc _x ,Δc _y ,Δc _z );

[0141] Among them, c _x c _y c _z These are the physical components of the absolute center coordinates of the target object in three dimensions, x _world y _world z _world These are the three-dimensional physical anchor point coordinate components of the highest-scoring voxel, Δc _x Δc _y Δc _z These represent the bounding box offsets for each dimension output by the network regression.

[0142] Optionally, the calculated absolute center coordinates and the predicted length, width, and height dimensions are packaged together to generate a positioning result describing the target's physical properties. This mechanism uses three-dimensional feature reference points as anchor points to regress relative offset values. Compared to regressing absolute coordinates from the entire map, this reduces the numerical search space of the regression algorithm and improves the convergence efficiency of numerical calculations and the accuracy of target positioning.

[0143] In some scenarios, when the highest target score in the calculated output is lower than the preset fault tolerance threshold, it is determined that no target entity matching the text feature description has been captured in the current visual input data, the coordinate offset addition operation is stopped, and prompt instruction data is generated for output.

[0144] In some embodiments, optional implementations of a global matching verification mechanism and multi-candidate target disambiguation data processing introduced in a cross-modal localization network are described. A second-stage attention calculation is also added to evaluate the overall feature relevance confidence of the scene, utilizing a large language model to resolve spatial referential ambiguity when multiple highly similar targets exist in the physical space. Accordingly, this embodiment can be:

[0145] Step 501: Using text features as the query and language-enhanced voxel representations as keys and values, perform the second-stage cross-modal attention calculation to evaluate the global matching confidence of the target object in the current scene.

[0146] We acquire the language-enhanced voxel representations corresponding to each physical anchor point, as well as the text features extracted from the front end. To determine whether physical entities that conform to natural language instructions actually exist within the 3D field of view captured by the current mobile terminal camera, we construct a feature query mapping relationship in reverse.

[0147] Specifically, text features are projected onto the query feature space using a linear transformation matrix, which is then used as the query matrix. Simultaneously, the language-enhanced voxel representation matrices within all non-zero space grids are projected onto the corresponding key and value feature spaces, forming the key and value matrices.

[0148] Furthermore, a second-stage cross-modal attention computation is performed to obtain the context vector of the overall field of view. This vector is then mapped to a numerical range of 0 to 1 using a multilayer perceptron and a nonlinear activation function, thereby evaluating the normalized scalar value of the output.

[0149] It should be understood that this scalar value represents the global matching confidence that the target object is present in the current scene. This scalar value characterizes the comprehensive probability parameter of the existence of the expected physical target in the three-dimensional space reconstructed from the two-dimensional image stream.

[0150] Step 502: When the global matching confidence is lower than the pre-configured matching confidence threshold, generate an interactive feedback instruction to prompt the user to adjust the viewing angle as the positioning result.

[0151] In other embodiments, when the global matching confidence is greater than or equal to the matching confidence threshold, the prediction of the target score and the generation of the localization result continue.

[0152] Optionally, the calculated global matching confidence score is compared with a pre-configured matching confidence score threshold, and a size comparison logical operation is performed.

[0153] For example, the pre-configured matching confidence threshold can be set to 0.65. If the comparison calculation result determines that the global matching confidence is <0.65, it is determined that the target object was not captured in the local observable field of view of the current camera, or that most of the physical surface of the target object is severely occluded by other geometric entities.

[0154] At this point, the exception handling branch logic for field-of-view adjustment is triggered. Preset trigger parameters are invoked to generate an interactive feedback command to prompt the user to adjust the viewing angle, serving as the positioning result. This interactive feedback command is specifically encoded as an audio control sequence or a rendering control protocol sequence for augmented reality interface coordinate points. After parsing this sequence, the mobile terminal outputs sensory feedback signals to the user, guiding the user to move the acquisition device to change the current camera pose matrix and acquire a new environmental scan image stream.

[0155] Step 503: When multiple voxels have target scores that exceed the pre-configured effective threshold and the multiple voxels are different in three-dimensional space, it is determined that there are multiple candidate targets and their three-dimensional spatial relationships in the current scene.

[0156] In the multi-candidate, multi-round interaction aimed at improving positioning accuracy, the predicted values ​​output by the feature refinement network are evaluated for their global distribution. The target score of all elements in the grid set is extracted, and the score of each voxel is compared with a pre-configured effective threshold.

[0157] In another example, the pre-configured effective threshold can be set to 0.8. Extract the voxel set indexes with scores ≥ 0.8. If the extracted set contains more than 1 voxels, further obtain the 3D physical anchor point coordinates of these extracted voxels in the world coordinate system. Obtain the 3D Euclidean distance between any two high-scoring voxels through matrix operations.

[0158] In another example, if the calculated 3D Euclidean spatial distance is greater than the pre-configured minimum clustering physical interval distance, the two voxels are determined to be independent physical entities in physical space. Under this data determination condition, it is determined that there are multiple candidate targets and their 3D spatial relationships in the current scene.

[0159] For example, in a unified grid coordinate matrix, it is determined that both non-connected location regions with index coordinates (12,24,8) and index coordinates (48,24,8) have entity physical feature responses that meet the basic semantic feature extraction conditions, thereby establishing a multi-candidate physical referential ambiguity state.

[0160] Step 504: Using a pre-built large language model, generate interactive query feedback instructions based on multiple candidate targets and their three-dimensional spatial relationships;

[0161] Accordingly, serialization operations are used to extract the parameters of the multi-candidate targets and their three-dimensional spatial relationships. The extracted high-dimensional physical coordinate tensors are converted into structured input character formats of natural language text and transmitted to the pre-built large language model inference nodes.

[0162] For example, the language model analyzes the geometric coordinate deviations of multiple extracted candidate targets in their relative physical spatial orientation distribution based on the weight parameters of its built-in pre-trained neural network. Using this feature deviation data as a basis, it generates an interactive query feedback instruction. This interactive query feedback instruction contains corresponding control logic to request the user to confirm the spatial location parameters of multiple similar feature entities.

[0163] Step 505: Obtain the supplementary text instructions input by the user based on the interactive query feedback instructions, and update the text features using the supplementary text instructions;

[0164] The mobile terminal's communication module receives interactive query feedback commands and converts them into visual or audible physical signals for the terminal interface. The user provides text or voice data containing identification attributes based on these physical signals. The mobile terminal's sensors collect this data sequence and transmit it back to the cloud server using a data stream protocol, thereby obtaining supplementary text commands containing predetermined orientation or texture constraints. These supplementary text commands are then input into a text encoding network to extract dimensionality-reduced additional semantic feature vectors.

[0165] Next, the extracted additional semantic feature vectors are concatenated with the original text features using tensor concatenation, and the text features are updated using concatenation dimensionality reduction. The updated text features retain the original basic retrieval intent while adding orientation condition parameters to eliminate spatial physical ambiguity.

[0166] Step 506: Based on the updated text features, re-execute the prediction depth probability distribution, generate three-dimensional voxel features, and perform cross-modal feature fusion until a unique localization result is generated.

[0167] Optionally, after obtaining the updated feature matrix, a loop reset process for the in-memory computation graph of the processing pipeline state machine is triggered. The initially stored query data nodes are then replaced with the updated text features.

[0168] Optionally, based on the updated text features, the matrix multiplication nodes of the deep prior network are re-introduced to perform the computation operation.

[0169] Optionally, based on the new semantic constraints, pixel-level correlation-gated feature extraction is performed again to calculate and generate the depth probability distribution parameter matrix of the redirected physical constraints.

[0170] Optionally, by re-performing the outer product projection and spatial grid aggregation operations on the matrix, a new three-dimensional voxel feature tensor is generated.

[0171] Next, the voxel sparsity decision parameters are substituted again, and a cross-modal feature dot product attention fusion operation is performed. The data feature processing pipeline will be configured to execute iteratively, gradually suppressing the feature response amplitude of interfering voxels in each round by injecting supplementary spatial semantic constraints from the user. When the computational condition is determined, and the target score of the determination spatial grid is ≥0.8, the iterative operation terminates.

[0172] Based on this, the center coordinate position parameters and bounding box size parameters corresponding to the unique high-resolution voxel are extracted, and the relative displacement coordinates are summed to generate an unambiguous positioning result pointing to the unique physical entity.

[0173] In a further embodiment, during the pre-training stage of the network model, the pass-through estimator approximation strategy and corresponding training scheme adopted to address the technical obstacle of gradient backpropagation truncation caused by discrete binary filtering masking operations are as follows:

[0174] During the end-to-end training phase, a multi-task joint loss function can be used to supervise and optimize all network parameters. The joint loss function is a weighted combination of the object classification loss and the 3D bounding box regression loss, specifically:

[0175] L=L _cls +λ×L _reg ;

[0176] In the formula, L _cls The binary cross-entropy loss corresponding to the output of the target scoring network is used to supervise the target score prediction of each voxel; L _reg The corresponding smooth L1 loss for the bounding box regression network output is used to supervise the regression accuracy of the 3D center coordinate offset and length, width and height dimensions, also known as the SmoothL1 regression loss; λ corresponds to the pre-configured loss weight coefficient, which is used to balance the optimization magnitude of the two sub-tasks, and its specific value can be determined through cross-validation.

[0177] Regarding training data, training samples can be obtained from publicly available 3D visual localization datasets, such as the 3D language reference localization dataset ScanRefer and the 3D natural language reference dataset NR3D. Each sample contains a multi-view RGB image sequence, the corresponding camera pose parameters, natural language description text, and the 3D bounding box annotation of the target object in the world coordinate system.

[0178] Step 601: In the backpropagation calculation of the model, a pass-through estimator strategy is adopted to replace the non-differentiable gradient of the binary screening mask with the gradient of a temperature-controllable smooth approximation function.

[0179] In the model training phase of this embodiment, it is necessary to calculate the partial derivatives of the objective function with respect to each weight parameter in the network to perform parameter updates. The binary filter mask generated using the indicator function has most gradients as zero. This zero-gradient characteristic physically truncates the gradient path from the localization loss function to the upstream feature extraction network.

[0180] Based on this, a pass-through estimator strategy is introduced to handle the gradient calculation process. In the forward propagation calculation path of the model, the original logic is maintained, and a discrete binary mask matrix is ​​generated by hard thresholding. Based on this, the actual three-dimensional voxel sparsification filtering operation is performed to maintain the high sparsity and computational efficiency of the forward inference calculation matrix.

[0181] When entering the backpropagation computation path of the model, the step function used in the forward pass is stripped away, and the gradient values ​​of the binary mask are forcibly replaced with the gradient values ​​of a temperature-controlled smooth approximation function. This replacement operation opens up the backpropagation channel for the error signal in the mask layer.

[0182] Step 602, the specific expression of the smooth approximation function is:

[0183] m=σ((r-τ _K ) / T);

[0184] In the formula, m corresponds to the output value of the smoothed approximation function, σ corresponds to the Sigmoid activation function, r corresponds to the smoothed relevance score, and τ _K For the confidence threshold applied to generate the binary filter mask, T corresponds to the pre-configured temperature parameter;

[0185] In other words, a smooth and everywhere differentiable function is used to approximate the step function. In practical implementation, this corresponds to the following calculation formula:

[0186] m=σ((r-τ _K ) / T);

[0187] Where m corresponds to the smoothed approximate value of the computational output, σ corresponds to the Sigmoid activation function, r corresponds to the smoothed correlation score of each voxel obtained from the previous calculation, and τ _K The confidence threshold generated in the preceding calculation corresponds to the pre-configured temperature parameter that sets the smoothness of the curve.

[0188] The local partial derivatives of the approximate function are derived using the chain rule, and the overall gradient is calculated using the gradient signals propagated back from subsequent layers. The calculation of the partial derivatives during backpropagation is achieved using the following formula:

[0189] G _r =G _v ×r×v _prime ×(1 / T)×m×(1-m);

[0190] In the formula, G _r G is the partial derivative of the loss function with respect to the smoothed correlation score r. _v v is the partial derivative of the loss function with respect to the sparse voxel features of the output. _prime The input consists of the initial voxel features before refinement by 3D convolution. In the formula, the m x (1-m) term represents the standard derivative structure of the smoothing activation function itself.

[0191] The above reflects the gradient components transmitted to the relevance score r via a smooth approximation function after bypassing the gradient truncation of the binary screening mask through a pass-through estimator strategy. Those skilled in the art will understand that in actual backpropagation implementations, gradients transmitted through the soft-weighted multiplication path can also be accumulated simultaneously, and the specific gradient calculation method can be adaptively adjusted according to the automatic differentiation implementation of the training framework.

[0192] Based on this, the specific value of the pre-configured temperature parameter T directly controls the sharpness of the gradient approximation curve. In this embodiment, the pre-configured temperature parameter T is set to 0.1. The specific value of the temperature parameter can be adaptively adjusted according to the balance requirements between training stability and gradient approximation accuracy.

[0193] Step 603: By smoothing the gradient of the approximate function, the gradient of the loss function of the localization task is made to bypass the gradient truncation restriction of the binary screening mask and propagate back to the upstream network for joint parameter optimization.

[0194] After substitutional differentiation and partial derivative multiplication, the gradient of the error loss function generated by the 3D localization regression network successfully passes through the non-differentiable voxel sparsity cutoff layer. The error signal continues to propagate backward to the front end of the network, updating the internal parameters of the preceding calculation modules in sequence.

[0195] Accordingly, the gradient signal is fed back to the correlation scoring network to correct the bilinear mapping matrix, further fed back to the gating network to adjust the pixel matching weight parameters, and fed back to the deep prior network to update the mapping weights from global semantics to the depth distribution.

[0196] It should be understood that this backpropagation mechanism establishes a closed-loop data transmission channel that runs through the spatial positioning network, the feature sparsification module, and the depth probability prediction module, so that the parameters of each processing step can be jointly iteratively optimized under the constraints of the 3D positioning physical deviation.

[0197] In other embodiments, the loss function for the localization task can also be a weighted combination of SmoothL1 regression loss and Intersection over Union (IoU) loss, which respectively constrains the target center coordinate offset and the envelope overlap of the three-dimensional bounding box.

[0198] Step 604: Calculate the voxel-level semantic relevance score for each voxel based on the cross-modal bilinear interaction between 3D voxel features and text features.

[0199] The voxel-level semantic relevance scores are smoothed using three-dimensional spatial coherence to obtain the smoothed relevance scores.

[0200] The smoothed correlation score is directly used as the soft correlation weight to perform soft weighted calculation of the three-dimensional voxel features, thus maintaining the differentiability of the entire forward calculation process.

[0201] In other scenarios, alternative pre-training implementations of pure soft mask computation are provided. When the network training environment lacks the underlying support to perform complex pass-through estimator computation, a processing branch that circumvents the binary mask generation logic is adopted. The extracted smoothed correlation score itself is directly used as a floating-point soft correlation weight parameter.

[0202] Scalar multiplication is performed on the full state of the 3D voxel features within the spatial grid, and the output is a voxel matrix that has undergone soft-weighting of features. This computation path only includes basic multiplication and addition operations, thus maintaining the differentiability of the entire forward computation graph and the reverse chained differentiation process.

[0203] Step 605: Sort all voxels in descending order according to their smoothed correlation scores, and extract only high-scoring voxels within the pre-configured retention range. Then, refine the features using a three-dimensional sparse convolution operation.

[0204] After completing the soft-weighted feature calculation, sparsification is still required to reduce the computational cost of subsequent 3D convolutional layers. Based on the numerical scale of the smoothed correlation scores, a one-dimensional descending sort operation is performed on the voxel data blocks corresponding to all grid coordinates.

[0205] Furthermore, the pre-configured retention threshold is read, and a subset of high-scoring voxels at the top of the sorted index is extracted sequentially. Low-scoring voxels at the tail that exceed the truncation range are removed, and the retained feature matrix subset is fed into subsequent network modules for feature refinement using 3D sparse convolution operations.

[0206] In this implementation, although the array sorting and truncation operation itself is not differentiable with respect to the spatial index, it only performs the physical location transfer and quantity filtering of data elements, without changing the internal floating-point values ​​of the voxel features participating in subsequent matrix operations. Because the feature matrix parameters input to the downstream network are generated by a pre-differentiable soft-weighted multiplication, the gradient of the localization error can still be losslessly backpropagated to each upstream network along the smoothed correlation score variable. Thus, the sparsity reduction of the number of voxels is achieved while ensuring the differentiability of training computation.

[0207] In other embodiments, specific technical solutions for addressing the problem of local occlusion from a single viewpoint by utilizing cross-frame temporal feature memory and aggregation mechanisms when processing continuous two-dimensional image streams are described below;

[0208] Step 701: Aggregate the 3D point features that fall within the same spatial grid to generate 3D voxel features;

[0209] During feature aggregation operations, a temporal state storage sequence is introduced. For the initial 3D voxel features acquired at the current moment, they are spatially aligned and accumulated with historical features stored in the historical moment memory queue. Accordingly, the camera intrinsic and extrinsic pose matrices for the current and historical moments are extracted. The relative rigid body transformation matrix from the historical spatial coordinate system to the current spatial coordinate system is calculated using matrix inversion and multiplication operations. This relative rigid body transformation matrix is ​​then used to perform a 3D affine transformation calculation on the historical feature space mesh, aligning the 3D features extracted at the historical moment to the unified physical coordinate space of the current moment.

[0210] After alignment calculation is completed, language-guided temporal accumulation calculation is performed on cross-frame features. Since the acquisition device is in continuous motion, performing indiscriminate temporal accumulation on all pixels would introduce background redundant parameters and increase the burden of matrix operations.

[0211] Based on this, text features are extracted as gating parameters. The semantic correlation between aligned historical voxel features and text features is calculated using cross-modal feature dot product operations, and the temporal cumulative gating value is calculated through a nonlinear activation function mapping. This temporal cumulative gating value is used to quantify and control the retention ratio of historical feature parameters transferred to the current time-stack feature tensor.

[0212] Specifically, the cross-frame temporal accumulation calculation is implemented using the following formula:

[0213] V _t =V _curr +g _t ×V _prev_aligned ;

[0214] Among them, V _t V represents the three-dimensional voxel features of the calculated output. _curr g represents the initial 3D voxel features calculated based on single-frame image aggregation at the current moment. _t V represents the time-series cumulative gate value of the calculated output. _prev_aligned This represents the historical feature tensor calculated using three-dimensional rigid body alignment.

[0215] For example, at a certain feature extraction time, the target object region in the current single-frame image is locally occluded, and the normalized norm of its corresponding initial 3D voxel feature vector is 0.2. Historical feature data from the previous time before occlusion is read from the memory queue, and the corresponding feature vector norm after coordinate affine alignment is 0.8. The temporal cumulative gate value of this aligned historical feature and text feature is calculated to be 0.9. Substituting the feature parameters into the cumulative formula and performing floating-point operations, the current voxel feature norm value is calculated to be 0.2 + 0.9 × 0.8 = 0.92.

[0216] Among them, relying on a high temporal cumulative gating value of 0.9, the target features with high signal-to-noise ratio in historical data frames are superimposed and supplemented into the voxel grid where spatial occlusion is currently occurring, which helps to make up for the lack of spatial feature data caused by the limitation of physical field of view in a single view.

[0217] As an optional implementation, a maximum historical frame sequence retention threshold is set in the memory queue management module. When the length of the stored historical feature array is greater than or equal to this retention threshold, a first-in-first-out (FIFO) eviction strategy is executed based on the increasing timestamp sorting parameter.

[0218] As another optional implementation, for spatial grid nodes whose calculated output time-series cumulative threshold value is less than a pre-configured lower limit threshold, the accumulation and transmission path of their historical feature data is directly blocked, forcing the historical feature components to be set to zero vectors. Through a dynamic pruning strategy, the hardware memory resources occupied by the time-series storage queue are further compressed, ensuring the real-time performance parameters of continuous streaming inference computation.

[0219] In this embodiment, a smartphone is used as the mobile terminal to describe the specific process of locating a predetermined target object in an unknown indoor environment, namely:

[0220] Optionally, the real-time sensing data stream on the device side is constructed by the user running a client program via a smartphone (mobile terminal). The program calls the phone's built-in RGB camera to capture real-time preview images, for example, at a resolution of 1080P / frame rate of 30fps, and simultaneously activates the built-in IMU sensor to record the acceleration and angular velocity data at the moment of each frame image acquisition.

[0221] Optionally, low-latency encoding and cloud synchronization: The mobile terminal uses a hardware encoder, such as H.265, to compress the video stream in real time. To ensure the accuracy of spatial positioning, each frame of image is encapsulated with its corresponding camera intrinsic parameters (focal length, principal point coordinates) and the pose matrix calculated by the IMU, and sent to the cloud server via a 5G network using a low-latency transmission protocol, such as WebRTC.

[0222] Optionally, after receiving the data stream in the cloud, the cloud-based 3D spatial feature mapping performs a spatial transformation of the image center on the graphics processing unit (GPU) cluster.

[0223] For example, an explicit spatial coding strategy can be adopted to discretely sample 2D visual features in the depth direction, and project them onto the three-dimensional world coordinate system in combination with camera intrinsic parameters to construct a three-dimensional feature volume in the current field of view, thus eliminating the dependence on pre-acquired point cloud data.

[0224] Optionally, multimodal large model command parsing and localization can be performed using cloud-based multimodal large models, such as the JM3D-LLM 3D multimodal large language model. When a user inputs a natural language command, such as locating the white mug on a black computer desk, the following operations are performed:

[0225] Extract semantic keywords and their spatial logical relationships from instructions in large models;

[0226] Perform cross-modal attention computation on semantic features and 3D feature volumes;

[0227] The positioning head reconstructs the center coordinates, length, width, height, and yaw angle of the target object in three-dimensional space.

[0228] Optionally, the interactive loop and real-time feedback cloud will transmit the calculated 3D bounding box coordinates and the guidance text generated by the large model, such as "A mug has been found for you, located 1.5 meters ahead," back to the mobile terminal via a downlink. The mobile terminal will then use AR to render the 3D positioning box in real time in the preview screen, achieving a visual feedback that blends the virtual and real worlds.

[0229] For example, when a user provides a vague description, such as "the cup for drinking water," the cloud-based multimodal big data model uses inference to discover that multiple cups exist in the scene, such as a plastic bottle and a mug. The big data model then sends a query request via the cloud: "Do you mean the blue plastic bottle or the white mug?" If the user answers, "The white one."

[0230] By combining the context of the previous dialogue, a multimodal fusion layer is used to further filter visual features and locate the three-dimensional position of the white mug. Supported by cloud computing power, this invention possesses stronger semantic understanding and logical reasoning capabilities than traditional edge-side models.

[0231] The optional embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solution of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A real-time 3D visual localization method based on multimodal fusion, characterized in that, include: Acquire a 2D image stream containing the target object, and a natural language query command describing the target object; Extract visual features from 2D image streams and extract textual features from natural language query commands; Based on visual features, predict the depth probability distribution of each pixel; Based on the depth probability distribution, visual features are enhanced to three-dimensional space and aggregated to generate three-dimensional voxel features; The three-dimensional voxel features are processed to obtain the target voxel features; Cross-modal feature fusion of text features and target voxel features is performed to locate the target object in three-dimensional space and output the localization result; The process involves cross-modal feature fusion of text features and target voxel features to locate the target object in 3D space and output the localization result. This includes: injecting spatial location information into the target voxel features using 3D sinusoidal position coding based on world coordinates to generate a position-aware 3D representation; performing a first-stage cross-modal attention calculation using the position-aware 3D representation as the query and text features as the key and value to obtain a language-enhanced voxel representation; predicting the target score using a target scorer network and predicting the 3D bounding box offset and size using a bounding box regression network for each language-enhanced voxel representation; and selecting the voxel with the highest target score and calculating its corresponding 3D bounding box offset and size along with the 3D physical anchor coordinates to generate the localization result.

2. The method according to claim 1, characterized in that, Extracting textual features from natural language query commands, including: Text features are obtained by directly encoding the features of natural language query commands using a text encoder. Alternatively, a large language model can be used to perform logical inference and ambiguity resolution on natural language query commands, outputting structured query text; a text encoder can then be used to encode the features of the structured query text to obtain text features.

3. The method according to claim 1, characterized in that, Based on depth probability distribution, visual features are uplifted to three-dimensional space and aggregated to generate three-dimensional voxel features, including: Obtain the camera's intrinsic parameter matrix and extrinsic pose matrix in real time; The visual features are multiplied by the depth probability distribution to generate three-dimensional point features at each discrete depth. By combining the camera's intrinsic and extrinsic pose matrices, 3D point features are projected onto a unified world coordinate system. Three-dimensional point features falling within the same spatial grid are aggregated to generate three-dimensional voxel features.

4. The method according to claim 1, characterized in that, Based on visual features, predict the depth probability distribution of each pixel, including: A deep prediction network is invoked to map visual features and obtain the visual basic distribution; Text features are processed using a deep prior network to obtain a global linguistic conditional prior distribution; Based on the cross-modal bilinear interaction between visual and textual features, the pixel-level correlation gating value corresponding to each pixel is calculated. Based on pixel-level correlation gating values, the visual baseline distribution and the linguistic conditional prior distribution are fused to obtain the depth probability distribution.

5. The method according to claim 3, characterized in that, After aggregating the 3D point features falling within the same spatial grid to generate 3D voxel features, the process also includes: A three-dimensional convolutional neural network is used to model the local spatial relationships of three-dimensional voxel features and output refined three-dimensional voxel features.

6. The method according to claim 1, characterized in that, The three-dimensional voxel features are processed to obtain the target voxel features, including: The full-state 3D voxel features are directly used as the target voxel features; Alternatively, the semantic correlation between 3D voxel features and text features can be calculated, and adaptive voxel sparsification can be performed accordingly to obtain the target voxel features in a sparse state.

7. The method according to claim 1, characterized in that, After performing the first-stage cross-modal attention computation to obtain the language-enhanced voxel representation, the process also includes: Using text features as queries and language-enhanced voxel representations as keys and values, a second-stage cross-modal attention computation is performed to evaluate the global matching confidence that the target object is contained in the current scene. When the global matching confidence score is lower than the matching confidence score threshold, an interactive feedback instruction is generated to prompt the user to adjust the viewing angle as the positioning result.

8. The method according to claim 1, characterized in that, Using location-aware 3D representations as queries and text features as keys and values, a first-stage cross-modal attention computation is performed to obtain language-enhanced voxel representations, including: Extracting corresponding attention output features based on cross-modal attention computation; The attention output features are summed with the position-aware 3D representation through residual connection, and the summed features are used as voxel representations for language enhancement.

9. A real-time three-dimensional vision positioning system based on multimodal fusion, characterized in that, include: At least one processor; Memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by a processor to implement the method described in any one of claims 1 to 8.

Images