Three-dimensional positioning method, program product, electronic device and storage medium

Abstract

The application provides a three-dimensional positioning method, a program product, an electronic device and a storage medium. The method comprises the following steps: obtaining a plurality of RGB image sequences and corresponding depth map sequences of multiple perspectives covering a to-be-positioned region from three-dimensional scene data of the to-be-positioned region; receiving a natural language text query instruction, respectively extracting a text embedding vector of the natural language text query instruction and a visual feature map of the RGB image, and generating a two-dimensional semantic confidence score map of multiple perspectives; calculating the visibility coefficient of a voxel under multiple perspectives by using the depth map sequence, and performing weighted back projection and spatial fusion on the two-dimensional semantic confidence score map by using the visibility coefficient to generate a three-dimensional language evidence body, and analyzing the three-dimensional language evidence body to output a three-dimensional coordinate of a target object in the to-be-positioned region. The zero-sample positioning, fine-grained semantic disambiguation and visibility-aware back projection and evidence fusion mechanism are realized, and the positioning robustness in a cluttered environment is enhanced.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a three-dimensional positioning method, a program product, an electronic device, and a storage medium. Background Technology

[0002] 3D visual localization refers to the technology of locating specific objects in three-dimensional space based on natural language descriptions, and it is one of the key capabilities for enabling robot-human interaction and autonomous operation. With the development of service robots and special-purpose robots, the primary solution for enabling robots to accurately understand human language commands and locate corresponding objects in three-dimensional space is through fully supervised learning methods for 3D localization. However, this method requires extensive annotation work, leading to low efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a three-dimensional positioning method, program product, electronic device, and storage medium to improve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide a three-dimensional localization method, comprising: acquiring RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the area to be localized from three-dimensional scene data of the area to be localized; receiving a natural language text query instruction, extracting the text embedding vector of the natural language text query instruction and the visual feature maps of the RGB images in the RGB image sequences of multiple viewpoints respectively; generating a two-dimensional semantic credibility score map of multiple viewpoints based on the semantic similarity between the text embedding vector and the visual feature maps of multiple viewpoints; converting the three-dimensional space of the area to be localized into a voxel grid, calculating the visibility coefficient of each voxel in the voxel grid using the depth map sequence and camera parameters, and using the visibility coefficient to perform weighted back projection and spatial fusion on the two-dimensional semantic credibility score map to generate a three-dimensional language evidence body; the visibility coefficient is used to characterize whether the voxel is occluded in the corresponding viewpoint; analyzing the three-dimensional language evidence body and outputting the three-dimensional coordinates of the target object in the area to be localized.

[0005] In the above implementation process, there is no need to use paired labeled data of 3D point cloud and text for training. Semantic features are extracted from 2D images and transferred to 3D space through geometric projection, achieving zero-shot 3D localization capability, reducing dependence on labeled data, and improving the model's generalization ability in new scenarios. This application's embodiments generate a 2D semantic credibility score map by calculating semantic similarity at the pixel level, which can accurately respond to fine-grained descriptions of color, state, and spatial relationships in language instructions, effectively solving the semantic ambiguity problem in complex scenes. By introducing a visibility coefficient to weightedly fuse multi-view semantic evidence, and using depth information to determine whether each voxel is occluded in each viewpoint, noise interference from occluded views is eliminated, improving localization accuracy in cluttered environments. It can be quickly deployed to new scenes without training and achieve accurate understanding and localization of complex language instructions.

[0006] Optionally, in this embodiment of the application, a two-dimensional semantic credibility score map of multiple perspectives is generated based on the semantic similarity between the text embedding vector and the visual feature maps of multiple perspectives, including:

[0007] Using the calculation formula for two-dimensional semantic credibility score maps, two-dimensional semantic credibility score maps from multiple perspectives are generated; the calculation formula for two-dimensional semantic credibility score maps includes:

[0008]

[0009] in, Indicates the first Coordinates from each perspective The semantic credibility score at the location, This represents the local visual feature vector corresponding to the coordinates. This is a text embedding vector.

[0010] In the above implementation, cosine similarity is used as the semantic matching metric. The text embedding vector is directly compared with the local visual feature vector at each pixel location, preserving fine-grained information about color, texture, and state in the language commands. This effectively improves the semantic ambiguity problem when multiple similar objects coexist in complex scenes. Furthermore, because this calculation is performed at the pixel level, high-resolution spatial information is preserved, allowing subsequent 3D fusion to utilize more refined semantic evidence.

[0011] Optionally, in this embodiment, the camera parameters include camera intrinsic parameters and camera extrinsic parameters. Calculating the visibility coefficient of a voxel at multiple viewpoints using the depth map sequence and camera parameters includes: using the camera intrinsic parameters, projecting the voxel onto the image coordinate system of the target viewpoint to obtain a projection point, and obtaining the depth value of the projection point in the depth map; using the camera extrinsic parameters, calculating the distance from the voxel to the camera optical center; if the absolute value of the difference between the distance and the depth value is less than a preset tolerance threshold, then the visibility coefficient value is determined to represent no occlusion; otherwise, the visibility coefficient value is determined to represent occlusion.

[0012] In the above implementation process, the visibility of each voxel is physically determined at each viewpoint using depth information, accurately identifying whether the voxel is occluded by other objects. No manual annotation or pre-training is required; the calculation is performed directly using depth data acquired by the sensor and camera parameters, maintaining the zero-sample characteristic of the entire method. Furthermore, by introducing a preset tolerance threshold, it can tolerate small errors in actual measurements, improving the robustness of the calculation process.

[0013] Optionally, in this embodiment of the application, a three-dimensional language evidence body is generated by weighted back projection and spatial fusion of a two-dimensional semantic credibility score map using a visibility coefficient. This includes: for each voxel, determining the visibility of the voxel under multiple viewpoints based on the visibility coefficient, and filtering out the target viewpoint of the voxel; weighted averaging the scores of the corresponding pixel positions in the two-dimensional semantic credibility score map corresponding to the target viewpoint to obtain the semantic fusion score of the voxel; generating a three-dimensional language evidence body based on the semantic fusion scores of multiple voxels; the three-dimensional language evidence body is used to characterize the semantic matching degree between each position in three-dimensional space and the natural language text query command.

[0014] In the above implementation process, the target viewpoint of each voxel is selected through visibility coefficients, ensuring that only semantic features of truly visible surfaces participate in the fusion. This physically eliminates noise interference from occluded viewpoints, improving the accuracy of semantic evidence fusion. A weighted average method is used to fuse the semantic credibility scores of multiple visible viewpoints, integrating semantic information from multiple perspectives. This effectively compensates for information loss caused by occlusion, insufficient lighting, etc., in a single viewpoint, enhancing robustness in complex environments. The generated 3D language evidence body meticulously depicts the semantic matching degree between each position in 3D space and the language command in the form of a voxel mesh, providing high-quality input for the precise localization of target objects. The entire construction process requires no 3D labeled data for training, directly utilizing features extracted from a pre-trained 2D vision-language model and depth information acquired by sensors, maintaining the zero-sample characteristic of the method. This allows the system to be quickly transferred to new scenes without additional training costs.

[0015] Optionally, in this embodiment of the application, obtaining RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the area to be located from the three-dimensional scene data of the area to be located includes: acquiring environmental data of the area to be located using an RGB-D camera, generating three-dimensional scene data based on the environmental data; and filtering RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the area to be located from consecutive frames of the three-dimensional scene data based on a preset sampling strategy; wherein the sampling strategy includes filtering based on pose changes and image sharpness changes between adjacent viewpoints.

[0016] In the above implementation process, 3D scene data is generated in real time using an RGB-D camera combined with SLAM technology, eliminating the need for pre-placed markers or external positioning equipment, thus reducing the complexity and cost of system deployment. A dual sampling strategy based on pose change and image sharpness is employed to eliminate redundant and blurry frames while covering the area to be located, reducing the amount of data required for subsequent processing and improving overall processing efficiency.

[0017] Optionally, in this embodiment of the application, the text embedding vector of the natural language text query instruction and the visual feature map of the RGB image in the RGB image sequence from multiple perspectives are extracted respectively, including: using a pre-trained two-dimensional visual language model to extract the text embedding vector of the natural language text query instruction and the visual feature map of the RGB image in the RGB image sequence from multiple perspectives respectively; wherein, the pre-trained two-dimensional visual language model adopts the CLIP model or the GLIP model.

[0018] In the above implementation process, a model pre-trained on large-scale data is used, eliminating the need for additional training or fine-tuning for 3D localization tasks. This avoids the high costs of collecting and labeling 3D point cloud-text pairing data, thus improving the training efficiency of the model. Since the pre-trained model has already learned rich visual and linguistic concepts and can understand natural language descriptions of open vocabularies (such as color, state, spatial relationships, etc.), transferring these concepts to 3D space enables the system to have good generalization ability when facing new scenes and objects, allowing it to adapt to different application environments without retraining.

[0019] Optionally, in this embodiment of the application, analyzing the three-dimensional language evidence body and outputting the three-dimensional coordinates of the target object in the region to be located includes: filtering the three-dimensional language evidence body using an adaptive threshold to obtain a filtered voxel region; performing three-dimensional connected component clustering on the filtered voxel region to extract multiple voxel clusters, and determining the target object from the multiple voxel clusters; calculating the geometric center of the voxel clusters of the target object, and generating the three-dimensional coordinates of the target object.

[0020] In the above implementation process, through adaptive threshold filtering, the system can dynamically adapt to the semantic matching score distribution under different scenarios, without the need for manually setting a fixed threshold, thus improving the accuracy and scene adaptability of filtering. Secondly, 3D connected component clustering can aggregate voxels belonging to the same object into a whole, effectively distinguishing different objects, avoiding interference from isolated noise points, and improving the robustness of target object extraction.

[0021] Secondly, embodiments of this application also provide a three-dimensional positioning device, comprising: a three-dimensional scene data module, used to acquire RGB image sequences covering multiple perspectives of the area to be positioned and corresponding depth map sequences from the three-dimensional scene data of the area to be positioned; a feature extraction module, used to receive natural language text query instructions, extract the text embedding vector of the natural language text query instructions, and the visual feature maps of the RGB images in the RGB image sequences of multiple perspectives; a credibility score module, used to generate a two-dimensional semantic credibility score map of multiple perspectives based on the semantic similarity between the text embedding vector and the visual feature maps of multiple perspectives; a three-dimensional language evidence body module, used to convert the three-dimensional space of the area to be positioned into a voxel grid, calculate the visibility coefficient of each voxel in the voxel grid using the depth map sequence and camera parameters, and use the visibility coefficient to perform weighted back projection and spatial fusion on the two-dimensional semantic credibility score map to generate a three-dimensional language evidence body; the visibility coefficient is used to characterize whether the voxel is occluded in the corresponding perspective; and a coordinate output module, used to analyze the three-dimensional language evidence body and output the three-dimensional coordinates of the target object in the area to be positioned.

[0022] Thirdly, embodiments of this application also provide a computer program product, including computer program instructions, which are executed by a processor to perform the method provided in the first aspect or any implementation thereof.

[0023] Fourthly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing computer program instructions, which are executed by the processor to perform the method provided in the first aspect or any implementation thereof.

[0024] Fifthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the method provided in the first aspect or any implementation thereof.

[0025] This application provides a 3D localization method, program product, electronic device, and storage medium. It eliminates the need for training with paired labeled data of 3D point clouds and text, extracts semantic features from 2D images, and transfers them to 3D space via geometric projection, achieving zero-shot 3D localization capabilities. This reduces reliance on labeled data and improves the model's generalization ability in new scenarios. The embodiments of this application generate a 2D semantic credibility score map by calculating semantic similarity at the pixel level, accurately responding to fine-grained descriptions of color, state, and spatial relationships in language instructions, effectively solving semantic ambiguity problems in complex scenes. By introducing a visibility coefficient to weightedly fuse multi-view semantic evidence and using depth information to determine whether each voxel is occluded in each viewpoint, noise interference from occluded views is eliminated, improving localization accuracy in cluttered environments. It can be quickly deployed to new scenes without training and achieves accurate understanding and localization of complex language instructions. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a three-dimensional positioning method provided in an embodiment of this application;

[0028] Figure 2 A flowchart of a zero-shot natural language 3D localization method based on multi-view language evidence fusion and 2D-VLM is provided for embodiments of this application;

[0029] Figure 3 This is a system structure diagram provided for an embodiment of this application.

[0030] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0033] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0034] 3D Visual Grounding (3DVG) aims to locate specific objects in three-dimensional space based on natural language descriptions, and is one of the core capabilities of embodied AI robots to achieve human-robot interaction and autonomous operation. With the rapid development of service robots and special-purpose robots, how to enable robots to accurately understand complex human language commands and find corresponding objects in three-dimensional space has become a research hotspot in academia and industry.

[0035] However, existing 3D visual positioning technologies mainly suffer from the following technical bottlenecks:

[0036] 1) Heavy reliance on expensive labeled data: Mainstream fully supervised methods typically require a large amount of expensive "3D point cloud-text" labeled data for training. This data collection is costly and time-consuming, and the trained models often struggle to adapt to unseen scenarios (such as sudden fire rescue scenes), exhibiting severe domain gap problems and extremely poor generalization ability.

[0037] 2) Insufficient semantic disambiguation capability: Traditional methods are mostly based on coarse-grained matching of object categories (e.g., detecting all "chairs" first and then making a selection). In complex unstructured scenes, there are often multiple objects with similar appearances (e.g., multiple fire extinguishers with similar shapes). Existing technologies have difficulty effectively handling instructions that include fine-grained descriptions such as state (e.g., "fallen down") and spatial relationships (e.g., "in the corner"), leading to localization failure.

[0038] 3) Single-view visual degradation and occlusion issues: In environments with smoke, insufficient lighting, or clutter, the reliability of object recognition from a single viewpoint will drop significantly. Existing multi-view fusion methods often lack effective geometric consistency constraints and cannot eliminate noise interference from occluded views during the fusion process, resulting in limited positioning accuracy.

[0039] The purpose of the embodiments provided in this application is to improve the shortcomings of existing 3D visual positioning technology, such as dependence on large-scale 3D labeled data, insufficient semantic disambiguation capability under complex instructions, and visual degradation and occlusion under single viewpoint. A zero-sample natural language 3D positioning method and system based on Language Evidence Body (LEV) is proposed.

[0040] 1) Eliminate 3D data dependency: By transferring the open-vocabulary semantic understanding capability of the pre-trained 2D vision-language model (2D-VLM) to 3D space, zero-sample localization of data without any 3D "point cloud-text" is achieved.

[0041] 2) Improve fine-grained semantic understanding: By utilizing pixel-level semantic credibility score maps, the system can accurately understand and respond to complex instructions that contain fine-grained descriptions such as color, texture, state (e.g., "falling down"), and spatial location, thus solving the semantic disambiguation problem.

[0042] 3) Enhanced robustness against occlusion and environmental conditions: By introducing strict geometric consistency constraints and a visibility-aware back-projection mechanism, noise from occluded viewpoints is dynamically eliminated using depth information, ensuring positioning accuracy in unstructured or extreme environments.

[0043] Please see Figure 1 The illustration shows a flowchart of a three-dimensional positioning method provided in an embodiment of this application. The three-dimensional positioning method provided in this application can be applied to electronic devices, which may include physical devices such as servers, PCs, tablets, or smartphones, or virtual devices such as virtual machines or containers. The electronic device can be a single device, a combination of multiple devices, or a cluster of a large number of devices. The three-dimensional positioning method may include:

[0044] Step S110: Obtain RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the area to be located from the 3D scene data of the area to be located.

[0045] Step S120: Receive natural language text query instructions, and extract the text embedding vector of the natural language text query instructions and the visual feature maps of RGB images in RGB image sequences from multiple perspectives.

[0046] Step S130: Generate a two-dimensional semantic credibility score map from multiple perspectives based on the semantic similarity between the text embedding vector and the visual feature maps from multiple perspectives.

[0047] Step S140: The three-dimensional space of the area to be located is transformed into a voxel mesh. For each voxel in the voxel mesh, the visibility coefficient of the voxel under multiple viewpoints is calculated using the depth map sequence and camera parameters. The visibility coefficient is then used to perform weighted back projection and spatial fusion on the two-dimensional semantic credibility score map to generate a three-dimensional language evidence body. The visibility coefficient is used to characterize whether the voxel is occluded under the corresponding viewpoint.

[0048] Step S150: Analyze the three-dimensional language evidence and output the three-dimensional coordinates of the target object in the area to be located.

[0049] In step S110, 3D scene data refers to a set of information that reflects the 3D spatial structure of the area to be located, obtained through sensor acquisition and reconstruction. For example, a robot equipped with an RGB-D depth camera can enter the area to be located. The RGB-D camera can simultaneously acquire color images and corresponding depth information. During the robot's movement, it uses Simultaneous Localization and Mapping (SLAM) technology to estimate the camera's position and orientation in real time, and gradually constructs a 3D point cloud or voxel map of the environment, i.e., 3D scene data.

[0050] After obtaining the 3D scene data, it is necessary to filter out multiple RGB image sequences and corresponding depth map sequences from them. Here, "viewpoint" refers to the angle from which the camera observes the scene from different positions and orientations. To reduce data redundancy while ensuring scene coverage, a preset sampling strategy can be used for key viewpoint filtering. The sampling strategy mainly includes two aspects: First, filtering is based on pose changes between adjacent frames. When the camera's translation distance exceeds a preset threshold (e.g., 0.5 meters) or rotation angle exceeds a preset threshold (e.g., 30 degrees), the frame is retained as a key viewpoint. Second, image sharpness is evaluated by using the Laplacian variance algorithm to detect image blurriness and automatically removing blurry frames below a preset sharpness threshold. These two filtering methods can be used in combination or individually. After the above filtering, a multi-view RGB image sequence and corresponding depth map sequence covering the area to be located are finally obtained. These data will serve as input for subsequent steps.

[0051] In step S120, the natural language text query instruction refers to the characteristics of the target object described by the user in everyday language, such as "a blue gas cylinder lying on its side by the window". This instruction is entered into the system in text form.

[0052] A pre-trained 2D visual language model is used to extract text embedding vectors and visual feature maps, respectively. A 2D visual language model is a neural network model pre-trained on large-scale image-text data, capable of mapping images and text to the same semantic feature space. Commonly used models include CLIP (Contrastive Language-Image Pre-trained Model) or GLIP (Grounded Language-Image Pre-trained Model). For example, a natural language text query command is input into the model's text encoder, and the output is a fixed-dimensional vector, i.e., a text embedding vector, which encodes the semantic information of the entire command.

[0053] Each RGB image is input into the model's image encoder, which outputs a visual feature map. The visual feature map is a two-dimensional matrix, where each location corresponds to a pixel region in the image. Each location contains a high-dimensional feature vector that encodes the visual semantic information of that region within the image. Through this step, text and image are transformed into the same semantic space, laying the foundation for subsequent semantic matching.

[0054] In step S130, the two-dimensional semantic credibility score map is a two-dimensional map with the same size as the original RGB image, where the value of each pixel represents the degree of semantic matching between the pixel location and the natural language text query instruction.

[0055] For each viewpoint's visual feature map, traverse each pixel position in the map and extract the corresponding local visual feature vector. Then, calculate the cosine similarity between this local visual feature vector and the text embedding vector obtained in step S120. Cosine similarity measures the similarity in direction between two vectors by calculating the cosine of the angle between them, with a value ranging from -1 to 1; a larger value indicates a closer semantic similarity. Use the calculated cosine similarity as the semantic credibility score for that pixel position. After performing the above operation on all pixel positions, a two-dimensional semantic credibility score map for that viewpoint is obtained. Since this calculation is performed at the pixel level, it can preserve fine-grained information about color, texture, and state in language instructions. For example, descriptions such as "blue" and "falling down" can obtain corresponding response regions in the score map.

[0056] In step S140, a voxel is short for volume pixel in three-dimensional space. In implementation, a bounding box is determined based on the spatial extent of the three-dimensional scene data; that is, the smallest cuboid capable of encompassing the entire area to be located. Then, according to a preset voxel resolution (e.g., 0.05 meters), the bounding box is divided into regularly arranged cubic units, each unit being called a voxel.

[0057] For each voxel, the visibility coefficient of that voxel under multiple viewpoints is calculated using a depth map sequence and camera parameters. Camera parameters include intrinsic and extrinsic parameters. The intrinsic parameters describe the imaging characteristics of the camera itself, while the extrinsic parameters describe the camera's position and orientation in the world coordinate system. The visibility coefficient is a binary quantity used to characterize whether the voxel is occluded at the corresponding viewpoint.

[0058] First, the camera's intrinsic parameters are used to project the voxel's 3D coordinates onto the image coordinate system of a certain viewpoint, obtaining the corresponding pixel coordinates. Then, the camera's extrinsic parameters are used to calculate the actual distance from the voxel to the camera's optical center. Simultaneously, the depth value at the same pixel coordinate is read from the depth map of that viewpoint; this depth value represents the distance from the camera to the actual surface point corresponding to that pixel. The actual distance and the depth value are compared. If the absolute value of the difference between the two is less than a preset tolerance threshold (e.g., 0.1 meters), it means that the voxel is located on a visible surface and is not occluded, and the visibility coefficient is set to 1; otherwise, it means that the voxel is occluded by other objects, and the visibility coefficient is set to 0.

[0059] Finally, the visibility coefficient is used to perform weighted back projection and spatial fusion on the 2D semantic credibility score map to generate a 3D language evidence body. The 3D language evidence body is a 3D array with the same dimensions as the voxel grid, where the value of each voxel represents the semantic matching degree between that spatial location and the natural language text query command. Specifically, for each voxel, only views with a visibility coefficient of 1 are selected, and the scores of the corresponding pixel positions in the 2D semantic credibility score map of these views are weighted and averaged to obtain the semantic fusion score for that voxel. After performing the above operation on all voxels, the complete 3D language evidence body is obtained. This process uses the visibility coefficient to eliminate noise interference from occluded views, ensuring that only the semantic features of truly visible surfaces participate in the fusion.

[0060] In step S150, the three-dimensional language evidence body is a three-dimensional probability density field. The higher the value of the voxel, the higher the semantic matching degree between the position and the language instruction.

[0061] First, an adaptive threshold is used to filter the 3D language evidence volume to mask background noise. The adaptive threshold can be calculated based on the statistical distribution of all voxel values ​​in the 3D language evidence volume. For example, a threshold can be set as a multiple of the standard deviation of the mean, filtering out voxels below the threshold and retaining high-response voxel regions.

[0062] Then, 3D connected component clustering is performed on the filtered high-response voxel regions. 3D connected component clustering refers to the operation of grouping spatially adjacent voxels with similar values ​​into the same cluster; commonly used methods include 26-neighborhood connectivity determination. Through clustering, voxels belonging to the same object can be grouped into a single voxel cluster, extracting voxel clusters corresponding to multiple candidate objects. The target object is then determined from multiple voxel clusters based on the response values; typically, the voxel cluster with the highest sum of response values ​​or the highest average response value is selected as the target object.

[0063] Finally, the geometric center of the voxel cluster of the target object is calculated. The geometric center can be obtained by averaging the 3D coordinates of all voxels within the voxel cluster, and this geometric center is output as the 3D coordinates of the target object. These 3D coordinates can be used for subsequent tasks such as robot navigation or grasping.

[0064] In the implementation of the above embodiments: Training is not required using paired labeled data of 3D point clouds and text. Semantic features are extracted from 2D images and transferred to 3D space through geometric projection, achieving zero-shot 3D localization capability. This reduces dependence on labeled data and improves the model's generalization ability in new scenarios. This application's embodiments generate a 2D semantic credibility score map by calculating semantic similarity at the pixel level, accurately responding to fine-grained descriptions of color, state, and spatial relationships in language instructions, effectively solving semantic ambiguity problems in complex scenes. By introducing a visibility coefficient to weightedly fuse multi-view semantic evidence, depth information is used to determine whether each voxel is occluded in each viewpoint, eliminating noise interference from occluded views and improving localization accuracy in cluttered environments. Rapid deployment to new scenes is possible without training, achieving accurate understanding and localization of complex language instructions.

[0065] Optionally, in this embodiment of the application, a two-dimensional semantic credibility score map of multiple perspectives is generated based on the semantic similarity between the text embedding vector and the visual feature maps of multiple perspectives, including:

[0066] Using the calculation formula for two-dimensional semantic credibility score maps, two-dimensional semantic credibility score maps from multiple perspectives are generated; the calculation formula for two-dimensional semantic credibility score maps includes:

[0067]

[0068] in, Indicates the first Coordinates from each perspective The semantic credibility score at the location, This represents the local visual feature vector corresponding to the coordinates. This is a text embedding vector.

[0069] Using this formula, the system can calculate the semantic similarity between each pixel location and the text instruction, thereby generating a fine-grained semantic credibility score map. Because this calculation is performed at the pixel level, it can preserve the responses of fine-grained information in the language instruction regarding color, texture, and state. For example, descriptions such as "blue" and "falling down" can be found in corresponding response areas in the score map.

[0070] In the implementation of the above embodiments: cosine similarity is used as a semantic matching metric, and the text embedding vector is directly compared with the local visual feature vector of each pixel position. This preserves the response of fine-grained information such as color, texture, and state in the language command. This effectively improves the semantic ambiguity problem when multiple similar objects coexist in complex scenes. Furthermore, since this calculation is performed at the pixel level, high-resolution spatial information is preserved, enabling subsequent 3D fusion to utilize more refined semantic evidence.

[0071] Optionally, in this embodiment, the camera parameters include camera intrinsic parameters and camera extrinsic parameters. The visibility coefficients of voxels at multiple viewpoints are calculated using depth map sequences and camera parameters, including:

[0072] Using the camera intrinsic parameters, the voxels are projected onto the image coordinate system of the target viewpoint to obtain the projection point, and the depth value of the projection point in the depth map is obtained.

[0073] Camera intrinsic parameters describe the internal imaging geometry of a camera, typically including focal length and principal point coordinates, and are used to map points in three-dimensional space onto a two-dimensional image plane. A voxel is a regular cubic unit obtained by discretizing the three-dimensional space of the area to be localized; each voxel has three-dimensional coordinates. Projection refers to using the camera intrinsic parameter matrix to transform the three-dimensional coordinates of the voxels into two-dimensional pixel coordinates in the target viewpoint image coordinate system through perspective projection transformation; the position corresponding to these coordinates is the projection point.

[0074] For each voxel and each viewpoint, the voxel's 3D coordinates are calculated using the camera's intrinsic matrix to obtain its pixel coordinates on the image plane. Since the calculated pixel coordinates may not be integers, methods such as bilinear interpolation can be used to obtain the precise projection point position. Then, based on the pixel coordinates of this projection point, the system reads the depth value at that location from the depth map of the corresponding viewpoint. The depth map is a 2D image registered with the RGB image, where the value of each pixel represents the distance from the camera to the actual surface point corresponding to that pixel. Through the above operations, the system obtains the depth value of the voxel's projection point at that viewpoint, which reflects the distance from the camera's optical center along the projection direction to the scene surface.

[0075] Using camera extrinsic parameters, the distance from the voxel to the camera optical center is calculated. If the absolute value of the difference between the distance and the depth value is less than the preset tolerance threshold, the visibility coefficient value is determined to indicate that the image is not occluded; otherwise, the visibility coefficient value is determined to indicate that the image is occluded.

[0076] Camera extrinsic parameters describe the camera's position and orientation in the world coordinate system. They typically include rotation matrices and translation vectors, used to transform points in the world coordinate system to the camera coordinate system. The camera optical center is the center point of the camera's optical system, located at the origin in the camera coordinate system. The distance from a voxel to the camera optical center is the Euclidean distance between the voxel's position in 3D space and the camera optical center. This distance can be calculated by transforming the voxel coordinates to the camera coordinate system using the camera extrinsic parameters.

[0077] First, the 3D coordinates of the voxel are transformed from the world coordinate system to the camera coordinate system using camera extrinsic parameters. Then, the depth value of the voxel in the camera coordinate system is calculated, which is the distance from the voxel to the camera optical center. The system compares the calculated distance with the depth value read from the depth map in step S310. If the absolute value of the difference is less than a preset tolerance threshold, it means that the voxel is located on the scene surface and is not occluded by other objects. In this case, the visibility coefficient is set to represent unoccluded (e.g., a value of 1). If the absolute value of the difference is greater than or equal to the preset tolerance threshold, it means that the voxel is occluded by other objects. In this case, the visibility coefficient is set to represent occluded (e.g., a value of 0). The preset tolerance threshold is used to tolerate depth measurement errors and projection errors. It is usually set to a small positive number, such as 0.05 meters or 0.1 meters, depending on the sensor accuracy and scene scale.

[0078] In the implementation of the above embodiments: the visibility of each voxel is physically determined at each viewpoint using depth information, accurately identifying whether the voxel is occluded by other objects. No manual annotation or pre-training is required; calculations are performed directly using depth data acquired by the sensor and camera parameters, maintaining the zero-sample characteristic of the entire method. Furthermore, by introducing a preset tolerance threshold, it can tolerate minor errors in actual measurements, improving the robustness of the calculation process.

[0079] As one implementation method, the visibility coefficient is expressed by a formula. The decision logic can be: calculate voxel points To the Euclidean distance of the optical center of a camera from a single perspective And query the projection point at the th Depth values ​​in a depth map from a single viewpoint ;

[0080] like ,but ;otherwise ;in This is the preset depth tolerance threshold.

[0081] Optionally, in this embodiment, a three-dimensional language evidence body is generated by weighted back projection and spatial fusion of the two-dimensional semantic credibility score map using visibility coefficients, including:

[0082] For each voxel, the visibility of the voxel under multiple viewpoints is determined based on the visibility coefficient, and the target viewpoint of the voxel is selected.

[0083] First, a list of visible viewpoints is established for each voxel. For the k-th viewpoint, if the voxel's visibility coefficient is a value indicating no occlusion (e.g., a value of 1), the viewpoint is marked as visible; if it is a value indicating occlusion (e.g., a value of 0), the viewpoint is marked as invisible. After traversing all viewpoints, the visibility of the voxel across multiple viewpoints is calculated based on the visibility coefficient of each viewpoint, i.e., the number of visible viewpoints is counted. Viewpoints with visibility coefficients indicating no occlusion are selected from all viewpoints and designated as target viewpoints for the voxel. Target viewpoints refer to the viewpoints that are actually visible in spatial location for the voxel; only the semantic information provided by these viewpoints is truly valid, while the semantic information of occluded viewpoints is excluded from subsequent fusion. Through this selection process, the system determines the set of valid viewpoints for each voxel to participate in subsequent fusion. The target viewpoint set represents the viewpoints from which the voxel is visible.

[0084] The semantic fusion score of a voxel is obtained by weighted averaging the scores of corresponding pixels in the two-dimensional semantic credibility score map corresponding to the target viewpoint.

[0085] For each target viewpoint, the system first projects the voxel's 3D coordinates onto the image coordinate system of that viewpoint using camera intrinsics, obtaining the corresponding pixel coordinates. Then, the score for that pixel position is read from the 2D semantic confidence score map of that viewpoint. Since the projected pixel coordinates may not be integers, the system can use methods such as bilinear interpolation to obtain the precise score for that position. For all target viewpoints of that voxel, the system obtains the corresponding semantic confidence scores for each viewpoint and then performs a weighted average. The weighted average can be an equal-weighted average, where the scores of all target viewpoints are summed and divided by the number of target viewpoints; or different weights can be set according to the confidence levels of different viewpoints. Through weighted averaging, the semantic fusion score of the voxel is obtained. This score integrates semantic evidence from multiple visible viewpoints, reflecting the overall semantic matching degree between the spatial location of the voxel and the natural language text query command.

[0086] A three-dimensional language evidence body is generated based on the semantic fusion scores of multiple voxels; the three-dimensional language evidence body is used to characterize the degree of semantic matching between each location in three-dimensional space and natural language text query instructions.

[0087] The 3D language evidence body is a 3D array with the same dimensions as the voxel grid, where each element corresponds to the semantic fusion score of a voxel. The voxel grid is a set of regularly arranged cubic cells obtained by discretizing the 3D space of the region to be localized, covering the entire region.

[0088] First, a 3D array of the same size as the voxel grid is initialized, with all elements initially set to 0. Then, the system iterates through each voxel in the region to be located, and for each voxel, the semantic fusion score is filled into the corresponding position in the 3D array. Once all voxels have been processed, the 3D array constitutes a complete 3D language evidence body.

[0089] In this three-dimensional language evidence volume, voxels with higher numerical values ​​indicate a higher degree of semantic matching between the spatial location and the natural language text query command, while voxels with lower numerical values ​​indicate a lower degree of matching. Through this three-dimensional language evidence volume, two-dimensional semantic evidence from multiple perspectives is accumulated in a geometrically consistent manner in three-dimensional space, forming a complete description of the possible locations of the target object in three-dimensional space, providing a foundation for subsequent target localization.

[0090] In the implementation of the above embodiments: the target viewpoint of each voxel is selected by visibility coefficient, ensuring that only the semantic features of the truly visible surface participate in the fusion, thus eliminating noise interference from occluded viewpoints at the physical level and improving the accuracy of semantic evidence fusion. A weighted average method is used to fuse the semantic credibility scores of multiple visible viewpoints, integrating semantic information from multiple perspectives. This effectively compensates for information loss caused by occlusion, insufficient lighting, etc., in a single viewpoint, enhancing robustness in complex environments. The generated 3D language evidence body meticulously depicts the semantic matching degree between each position in 3D space and the language instruction in the form of a voxel mesh, providing high-quality input for the precise positioning of the target object. The entire construction process requires no 3D labeled data for training, directly utilizing features extracted from a pre-trained 2D vision-language model and depth information acquired by sensors, maintaining the zero-sample characteristic of the method and enabling the system to quickly migrate to new scenes without additional training costs.

[0091] Optionally, in this embodiment of the application, obtaining an RGB image sequence covering multiple viewpoints of the area to be located and a corresponding depth map sequence from the 3D scene data of the area to be located includes:

[0092] Environmental data of the area to be located is collected by an RGB-D camera, and 3D scene data is generated based on the environmental data.

[0093] Environmental data of the area to be localized is acquired using an RGB-D camera, and 3D scene data is generated based on this environmental data. An RGB-D camera is a sensor capable of simultaneously acquiring color RGB images and depth images. The RGB images provide color texture information of the scene, while the depth images provide distance information from each pixel to the camera. Environmental data refers to the color image frames, depth image frames, and synchronously recorded camera pose information continuously acquired by the RGB-D camera as the robot moves within the area to be localized.

[0094] For example, a robot equipped with an RGB-D camera can be controlled to enter the area to be localized, with the camera collecting environmental data at a fixed frequency. Simultaneous Localization and Mapping (SLAM) technology is used to process this data in real time: on the one hand, the camera's position and orientation (i.e., pose) at each moment is estimated; on the other hand, the depth data collected at different moments are registered and fused to gradually construct a 3D point cloud map or voxel map of the area to be localized. This map is the 3D scene data. The 3D scene data completely records the spatial geometry of the area to be localized, including the 3D coordinate information of each point on the object's surface, providing a spatial reference for subsequent viewpoint selection and projection fusion.

[0095] Based on a preset sampling strategy, RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the area to be located are filtered from consecutive frames of 3D scene data; the sampling strategy includes filtering based on pose changes and image sharpness changes between adjacent viewpoints.

[0096] Based on a preset sampling strategy, RGB image sequences and corresponding depth map sequences covering multiple viewpoints within consecutive frames of 3D scene data are selected. A consecutive frame refers to the original image sequence captured by an RGB-D camera; each frame contains one RGB image, one depth image, and the corresponding camera pose.

[0097] Because there is a lot of redundancy in consecutive frames (e.g., the content of adjacent frames is highly similar when the robot is stationary or moving slowly), it is necessary to filter out representative and high-quality keyframes. The preset sampling strategy includes two methods: First, filtering is based on pose changes between adjacent viewpoints. This involves calculating the translation distance and rotation angle of the camera between adjacent frames. When the translation distance exceeds a preset threshold (e.g., 0.5 meters) or the rotation angle exceeds a preset threshold (e.g., 30 degrees), the current frame is retained as a key viewpoint. Second, filtering is based on changes in image sharpness. Specifically, the Laplacian variance algorithm is used to calculate the sharpness score of each RGB image frame, and blurry frames with scores below a preset sharpness threshold are automatically removed. After these two filtering steps, a set of key viewpoints is obtained. Each key viewpoint corresponds to one RGB image, one depth image, and camera intrinsic and extrinsic parameters. These data constitute a sequence of RGB images and corresponding depth maps covering multiple viewpoints of the area to be localized.

[0098] In the implementation of the above embodiments: 3D scene data is generated in real time using an RGB-D camera combined with SLAM technology, eliminating the need for pre-placed markers or external positioning equipment, thus reducing the complexity and cost of system deployment. A dual sampling strategy based on pose change and image sharpness is employed, eliminating redundant and blurry frames while covering the area to be located, reducing the amount of data required for subsequent processing and improving overall processing efficiency.

[0099] Furthermore, in an optional embodiment, the step of constructing a three-dimensional language evidence body (LEV) by weighted back-projection and spatial fusion of the two-dimensional semantic credibility score using the visibility coefficient via a formula includes: defining arbitrary voxel points in three-dimensional space. Its fusion score in the three-dimensional language evidence body The calculation formula is as follows:

[0100]

[0101] in, This represents the total number of sampled viewpoints; Indicates voxel points Projected to the Projection function of two-dimensional pixel coordinates from a viewpoint; For the first Semantic credibility score map sampling values ​​from each perspective; The visibility coefficient is a binary representation used to characterize voxel points. In the Is it obstructed from any angle?

[0102] Optionally, in this embodiment of the application, the text embedding vector of the natural language text query instruction and the visual feature maps of the RGB images in the RGB image sequence from multiple perspectives are extracted, including:

[0103] Using a pre-trained two-dimensional visual language model, text embedding vectors of natural language text query commands and visual feature maps of RGB images in RGB image sequences from multiple perspectives are extracted. The pre-trained two-dimensional visual language model adopts either the CLIP model or the GLIP model.

[0104] A pre-trained 2D visual language model refers to a neural network model pre-trained on large-scale image-text pairing data. This model can map images and text to the same high-dimensional semantic feature space. The model can include CLIP or GLIP models. CLIP (Contrastive Language-Image Pre-trained) models are trained through contrastive learning, making matched image-text pairs closer in the feature space and mismatched pairs farther apart. GLIP (Grounded Language-Image Pre-trained) models add object detection training to CLIP, enabling better association between words in text and regions in images. A text embedding vector is a fixed-dimensional numerical vector output from the model's text encoder after a natural language text query command is input. This numerical vector encodes the semantic information of the entire command.

[0105] A visual feature map is a two-dimensional matrix output from the image encoder of a model after an RGB image is input. Each position in the matrix corresponds to a local region in the image, and each position contains a high-dimensional feature vector that encodes the visual semantic information of that region. For example, a natural language text query command (such as "a blue gas cylinder lying on its side by the window") can be received from a user and input into a CLIP or GLIP text encoder to obtain a text embedding vector. Simultaneously, each selected RGB image is input into the same model's image encoder to obtain a corresponding visual feature map. These feature maps preserve the spatial resolution of the image, providing a foundation for subsequent pixel-level semantic matching.

[0106] In the implementation of the above embodiments: using a model pre-trained on large-scale data eliminates the need for additional training or fine-tuning for 3D localization tasks, avoiding the high costs of collecting and labeling 3D point cloud-text pairing data, and improving model training efficiency. Since the pre-trained model has already learned rich visual and linguistic concepts and can understand natural language descriptions of open vocabularies (such as color, state, spatial relationships, etc.), transferring these concepts to 3D space enables the system to have good generalization ability when facing new scenes and objects, adapting to different application environments without retraining.

[0107] Optionally, in this embodiment of the application, analyzing the three-dimensional language evidence and outputting the three-dimensional coordinates of the target object in the region to be located includes:

[0108] An adaptive threshold is used to filter the 3D language evidence volume to obtain the filtered voxel region.

[0109] An adaptive threshold is used to filter the 3D language evidence body, resulting in a filtered voxel region. The 3D language evidence body is a 3D array, where the value of each voxel represents the semantic matching degree between that spatial location and the natural language text query command; a higher value indicates a higher matching degree. The adaptive threshold is a filtering threshold dynamically calculated based on the statistical distribution of all voxel values ​​in the 3D language evidence body, rather than a pre-fixed constant.

[0110] In its implementation, the system first calculates the mean μ and standard deviation σ of all voxel values ​​in the 3D language evidence body. Then, it sets a threshold T = μ + k·σ, where k is a sensitivity coefficient (e.g., 2.0) to control the strictness of the filtering. Subsequently, the system iterates through each voxel in the 3D language evidence body, filtering out voxels with values ​​below the threshold T (marking them as background or ignoring them), and retaining only voxels with values ​​greater than or equal to the threshold T. These retained voxels constitute the filtered voxel region, which contains only spatial locations with a high degree of semantic matching to the natural language text query command, while low-matching background noise is effectively masked, thus reducing interference in subsequent processing.

[0111] Three-dimensional connected component clustering is performed on the filtered voxel regions to extract multiple voxel clusters, and the target object is determined from the multiple voxel clusters.

[0112] Three-dimensional connected component clustering is performed on the filtered voxel regions to extract multiple voxel clusters, and the target object is identified from these clusters. Three-dimensional connected component clustering refers to the process of grouping spatially adjacent voxels with similar numerical values ​​into the same category.

[0113] The implementation involves iterating through each voxel in the filtered voxel region and using a 26-neighbor connectivity criterion. This means checking the connectivity of a voxel with its 26 neighboring voxels (up, down, left, right, front, back, and diagonally), grouping connected voxels into the same voxel cluster. Each voxel cluster represents a contiguous spatial region, typically corresponding to a candidate object in the scene. After clustering, multiple voxel clusters are obtained.

[0114] The target object is then identified from these voxel clusters. This can be done by selecting the cluster containing the most voxels or by selecting the cluster with the highest average semantic fusion score. For example, the system can calculate the sum or average of the semantic fusion scores of all voxels within each voxel cluster and determine the voxel cluster with the highest score as the voxel cluster corresponding to the target object.

[0115] Calculate the geometric center of the voxel clusters of the target object and generate the three-dimensional coordinates of the target object.

[0116] The geometric center refers to the centroid of a voxel cluster in three-dimensional space, representing the approximate spatial location of the target object. In the implementation, it is assumed that the target object's voxel cluster contains N voxels, and the three-dimensional coordinates of each voxel are (X1, Y1, Z1), (X2, Y2, Z2), ..., (X...). N , Y N Z N The system calculates the average coordinates of all voxels in the X, Y, and Z directions, i.e., the X coordinate of the geometric center is (X1 + X2 + ... + X...).N ) / N, the Y coordinate is (Y1+ Y2+…+Y N ) / N, Z coordinate is (Z1+Z2+…+Z N The three average values ​​are calculated as () / N. These three values ​​are combined to form a three-dimensional coordinate point, which is the geometric center of the target object. The system outputs this geometric center as the three-dimensional coordinates of the target object, which can be directly used for downstream tasks such as robot navigation and grasping. For example, in a fire rescue scenario, the robot can plan a path based on the output three-dimensional coordinates to reach the location of the target object and perform rescue operations.

[0117] In the implementation of the above embodiments: Through adaptive threshold filtering, the system can dynamically adapt to the semantic matching score distribution under different scenarios, without the need for manually setting a fixed threshold, thus improving the accuracy and scene adaptability of filtering. Secondly, 3D connected component clustering can aggregate voxels belonging to the same object into a whole, effectively distinguishing different objects, avoiding interference from isolated noise points, and improving the robustness of target object extraction.

[0118] Please see Figure 2 The flowchart shown is a zero-sample natural language 3D localization method based on multi-view language evidence fusion and 2D-VLM provided in an embodiment of this application.

[0119] Natural language command input refers to the user inputting descriptive commands for the target object in natural language, serving as the semantic query input for the entire method. Environmental data of the area to be located is collected using sensors such as RGB-D cameras, and 3D scene data is reconstructed in real time using SLAM technology.

[0120] Natural language commands are input into a pre-trained 2D vision-language model (2D-VLM, such as CLIP or GLIP), and text embedding vectors are extracted. These vectors encode the global semantic information of the commands. Based on a preset sampling strategy (pose change threshold and image sharpness threshold), multi-view RGB image sequences covering the entire scene and corresponding depth map sequences are selected from the 3D scene data and used as input for visual features.

[0121] Calculate the pixel-level cosine similarity between the text embedding vector and the visual feature maps of images from various perspectives, and generate multiple two-dimensional semantic credibility score maps. The pixel value on each map represents the degree of semantic matching between the location and the instruction.

[0122] The three-dimensional space is voxelized, and the visibility coefficient of each voxel under each viewpoint is calculated using camera parameters and depth maps. The semantic credibility scores of the visible viewpoints are then weighted and back-projected and spatially fused to construct a three-dimensional language evidence body, where the value of each voxel represents the semantic matching strength between the spatial location and the instruction.

[0123] Adaptive threshold filtering and three-dimensional connected component clustering are performed on the three-dimensional language evidence body to extract the voxel clusters corresponding to the target object, calculate their geometric centers, and finally output the three-dimensional coordinates of the target object.

[0124] Please see Figure 3 The diagram shown is a system structure diagram provided in an embodiment of this application.

[0125] Figure 3 This paper illustrates the three-layer architecture and data flow of the 3D positioning system in this embodiment. The system is integrated onto a rescue robot or mobile platform. In the perception input layer, an RGB-D camera acquires image sequences and depth maps. The Simultaneous Localization and Mapping (SLAM) module estimates the pose trajectory and reconstructs the 3D scene in real time, while simultaneously receiving natural language commands (such as "find the blue gas cylinder"). The feature processing and fusion layer is the core algorithm: first, a pre-trained 2D-VLM (such as CLIP) is used to extract cross-modal features from the image sequences and natural language commands, calculating cross-modal similarity to generate a pixel-level semantic credibility score map (heatmap); then, combining the depth map, camera pose, and coordinate system transformation (world coordinate system), visibility verification (occlusion culling) is performed on each voxel, and a 3D language evidence body (LEV) is constructed through back-projection and Bayesian fusion. The decision and landing layer estimates the target position based on the 3D language evidence body, outputs 3D coordinates, plans a navigation path, and ultimately drives the robot to execute actions, completing the closed loop from semantic understanding to physical space positioning and navigation.

[0126] The beneficial effects of the embodiments of this application include, but are not limited to:

[0127] Zero-shot feature: The entire process does not require any 3D point cloud-text data for training. It directly utilizes the massive general knowledge of 2D-VLM, has a strong scene transfer capability, is plug-and-play, and is particularly suitable for emergency rescue scenarios where data is scarce.

[0128] Fine-grained semantic understanding capability: By generating pixel-level semantic credibility score maps, the model can respond to adjectives and state words such as "blue" and "near the door", effectively solving the semantic disambiguation problem in complex scenarios.

[0129] Robust anti-occlusion capability: The system introduces a visibility perception fusion mechanism and uses depth information to dynamically remove erroneous projections of occluded viewpoints, enabling the system to maintain high-precision positioning performance even in cluttered environments.

[0130] The advantages described above in this application can have a positive effect on the practical application of robots:

[0131] 1) Provide key technical support for fire rescue and post-disaster search and rescue: enable rescue robots to quickly and accurately locate rescue targets (such as gas cylinders, wounded people, etc.) through natural language commands in extreme environments filled with smoke and obstacles.

[0132] 2) Provides efficient interaction methods for human-machine collaboration with embodied intelligence: It lowers the threshold for robot operation, enabling non-professionals to command robots to complete complex three-dimensional spatial tasks through natural language, thereby improving the efficiency of emergency response.

[0133] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and specific mathematical formulas.

[0134] Implementation Example: Target Localization in an Indoor Fire Rescue Scenario This embodiment demonstrates how to use the method described in this invention to guide a robot to locate specific rescue supplies in a simulated indoor fire scene without prior training.

[0135] Step 1: The data acquisition and preprocessing system is integrated into the emergency rescue robot terminal. After the robot enters the fire scene, its onboard multimodal perception module (including an RGB-D camera and LiDAR) begins operation. Utilizing SLAM (Simultaneous Localization and Mapping) technology, it reconstructs the sparse point cloud or voxel grid of the current environment in real time and records the robot's pose trajectory at different times. The system acquires data covering the entire scene based on a preset sampling strategy (such as keyframe selection). Frame RGB image and its corresponding depth map .

[0136] Step 2: Generating a Semantic Trust Score Map. The system receives natural language text query instructions from the command center. (For example: "Find the blue gas cylinder lying on its side by the window." Use a pre-trained CLIP model as a 2D-VLM to extract features.) For each frame of the image within the field of view Calculate its feature map Calculate pixel points Semantic credibility score :

[0137]

[0138] The score chart This visually reflects which pixel areas highly respond to semantic features such as "blue," "window edge," and "gas cylinder" from the current perspective. The higher the responsivity, the closer its pixel value is to 1.

[0139] Step 3: Construct the 3D Language Evidence Volume (LEV) and initialize a 3D tensor the same size as the scene bounding box. The scene bounding box is defined as the smallest three-dimensional cuboid space that encloses the region to be localized. It is determined by traversing the extreme values ​​of all valid coordinate points in the point cloud map constructed in real time by SLAM. To determine the boundary range, and in conjunction with the preset voxel resolution. Discretize the space into a voxel grid (e.g., 0.05m), setting the initial score of all voxels to 0. Iterate through each voxel in the space. Through the camera intrinsic parameter matrix and extrinsic parameter matrix Project it onto the first Frame image coordinate system:

[0140]

[0141] Visibility testing: To prevent voxels behind walls or inside objects from being incorrectly assigned values, the Z-buffer algorithm is used to verify voxels. Is it obstructed by an object in front? Calculate voxels. Distance to camera and compare it with the depth value at that location on the depth map. Compare. If ( If a depth tolerance (e.g., 0.1 meters) is set, then the voxel is determined to be visible from the current viewpoint, and the visibility coefficient is set accordingly. ;otherwise If visible, then use bilinear or trilinear interpolation to obtain the fractional plot. The value is updated and the voxel is updated. Fusion score:

[0142]

[0143] in To prevent minute amounts with a denominator of zero, this process accumulates geometrically consistent two-dimensional semantic evidence from multiple perspectives in three-dimensional space.

[0144] Step 4: 3D target analysis and localization. After full-view fusion, A three-dimensional probability density field is formed. The system processes this field using an adaptive threshold and calculates the mean of the global score. with standard deviation Set the filter threshold to ( (The sensitivity coefficient) is used to dynamically mask background noise. Subsequently, a three-dimensional connected component clustering algorithm is used to extract the voxel cluster with the largest sum of response values ​​as candidate targets, and the geometric center coordinates of the target are determined by calculating the arithmetic mean of the coordinates of all voxels within this connected component. Simultaneously, the extreme value intervals of the connected component in the three axes are extracted to construct a 3D bounding box that encloses the target object.

[0145] Step 5: Cooperative navigation: Positioning output coordinates The target will be sent to the downstream cooperative navigation module via the ROS (Robot Operating System) communication interface. The robot, combined with the real-time constructed 3D scene map (3DSG), will plan an optimal path to avoid obstacles and reach the "fallen blue gas cylinder" to complete the rescue mission.

[0146] Through the above implementation methods, the present invention can accurately understand and locate objects with complex attribute descriptions without pre-training samples, verifying the effectiveness and robustness of the language evidence fusion method.

[0147] This application provides a three-dimensional positioning device, including:

[0148] The 3D scene data module is used to obtain RGB image sequences and corresponding depth map sequences covering multiple perspectives of the area to be located from the 3D scene data of the area to be located.

[0149] The feature extraction module is used to receive natural language text query instructions, extract the text embedding vector of the natural language text query instructions, and the visual feature maps of RGB images in RGB image sequences from multiple perspectives.

[0150] The credibility score module is used to generate two-dimensional semantic credibility score maps from multiple perspectives based on the semantic similarity between the text embedding vector and the visual feature maps from multiple perspectives.

[0151] The 3D language evidence module is used to transform the 3D space of the region to be located into a voxel mesh. For each voxel in the voxel mesh, the visibility coefficient of the voxel under multiple viewpoints is calculated using the depth map sequence and camera parameters. The visibility coefficient is then used to perform weighted back projection and spatial fusion on the 2D semantic credibility score map to generate the 3D language evidence. The visibility coefficient is used to characterize whether the voxel is occluded under the corresponding viewpoint.

[0152] The coordinate output module is used to analyze the three-dimensional language evidence and output the three-dimensional coordinates of the target object in the area to be located.

[0153] Optionally, in this embodiment of the application, the three-dimensional positioning device and the credibility score module are specifically used to generate two-dimensional semantic credibility score maps from multiple perspectives using the calculation formula of the two-dimensional semantic credibility score map; the calculation formula of the two-dimensional semantic credibility score map includes:

[0154]

[0155] in, Indicates the first Coordinates from each perspective The semantic credibility score at the location, This represents the local visual feature vector corresponding to the coordinates. This is a text embedding vector.

[0156] Optionally, in this embodiment of the application, the three-dimensional positioning device and the three-dimensional language evidence module are specifically used to use camera intrinsic parameters to project voxels onto the image coordinate system of the target viewpoint to obtain projection points, and obtain the depth value of the projection points in the depth map; use camera extrinsic parameters to calculate the distance from the voxel to the camera optical center, and if the absolute value of the difference between the distance and the depth value is less than a preset tolerance threshold, then the visibility coefficient value is determined to represent no occlusion, otherwise the visibility coefficient value is determined to represent occlusion.

[0157] Optionally, in this embodiment of the application, the three-dimensional positioning device and the three-dimensional language evidence module are specifically used to determine the visibility of each voxel under multiple viewpoints based on the visibility coefficient, and filter out the target viewpoint of the voxel; to perform a weighted average of the scores of the corresponding pixel positions in the two-dimensional semantic credibility score map corresponding to the target viewpoint to obtain the semantic fusion score of the voxel; to generate a three-dimensional language evidence body based on the semantic fusion scores of multiple voxels; the three-dimensional language evidence body is used to characterize the semantic matching degree between each position in three-dimensional space and the natural language text query command.

[0158] Optionally, in this embodiment of the application, the three-dimensional positioning device and the three-dimensional scene data module are specifically used to collect environmental data of the area to be positioned through an RGB-D camera, generate three-dimensional scene data based on the environmental data, and filter RGB image sequences and corresponding depth map sequences covering multiple perspectives of the area to be positioned from the continuous frames of the three-dimensional scene data based on a preset sampling strategy; wherein, the sampling strategy includes filtering based on pose changes and image sharpness changes between adjacent perspectives.

[0159] Optionally, in the embodiments of this application, the three-dimensional positioning device and the feature extraction module are specifically used to extract the text embedding vector of the natural language text query instruction and the visual feature map of the RGB image in the RGB image sequence from multiple perspectives using a pre-trained two-dimensional visual language model; wherein the pre-trained two-dimensional visual language model adopts the CLIP model or the GLIP model.

[0160] Optionally, in the embodiments of this application, the three-dimensional positioning device and the coordinate output module are specifically used to filter the three-dimensional language evidence body using an adaptive threshold to obtain a filtered voxel region; perform three-dimensional connected component clustering on the filtered voxel region to extract multiple voxel clusters, and determine the target object from the multiple voxel clusters; calculate the geometric center of the voxel clusters of the target object, and generate the three-dimensional coordinates of the target object.

[0161] It should be understood that this device corresponds to the above-described three-dimensional positioning method embodiment and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0162] Please see Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0163] Figure 4 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a server or PC, or a virtual device, such as a virtual machine or virtualization container. Furthermore, electronic device 300 is not limited to a single device; it can be a combination of multiple devices or a cluster of numerous devices.

[0164] This application also provides a storage medium storing a computer program, which is executed by a processor to perform the above-described method.

[0165] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0166] This application also provides a computer program product, including computer program instructions, which are executed by a processor to perform the method described above.

[0167] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, given the several embodiments provided in this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0168] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0169] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A three-dimensional positioning method, characterized in that, include: From the 3D scene data of the area to be located, obtain RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the area to be located; Receive natural language text query instructions, and extract the text embedding vector of the natural language text query instructions and the visual feature maps of the RGB images in the RGB image sequence from multiple perspectives; Based on the semantic similarity between the text embedding vector and the visual feature maps from multiple perspectives, a two-dimensional semantic credibility score map from multiple perspectives is generated. The three-dimensional space of the region to be located is transformed into a voxel mesh. For each voxel in the voxel mesh, the visibility coefficient of the voxel under multiple viewpoints is calculated using the depth map sequence and camera parameters. The visibility coefficient is then used to perform weighted back projection and spatial fusion on the two-dimensional semantic credibility score map to generate a three-dimensional language evidence body. The visibility coefficient is used to characterize whether the voxel is occluded under the corresponding viewpoint. The three-dimensional language evidence is analyzed, and the three-dimensional coordinates of the target object in the region to be located are output.

2. The method according to claim 1, characterized in that, Based on the semantic similarity between the text embedding vector and the visual feature maps from multiple perspectives, a two-dimensional semantic credibility score map from multiple perspectives is generated, including: Using the calculation formula for two-dimensional semantic credibility score maps, two-dimensional semantic credibility score maps from multiple perspectives are generated; the calculation formula for the two-dimensional semantic credibility score maps includes: in, Indicates the first Coordinates from each perspective The semantic credibility score at the location, This represents the local visual feature vector at the location corresponding to the coordinates. The text embedding vector is denoted as .

3. The method according to claim 1, characterized in that, The camera parameters include camera intrinsic and camera extrinsic parameters. The visibility coefficients of the voxels at multiple viewpoints are calculated using the depth map sequence and camera parameters, including: Using the camera intrinsic parameters, the voxel is projected onto the image coordinate system of the target viewpoint to obtain the projection point, and the depth value of the projection point in the depth map is obtained. Using the camera extrinsic parameters, the distance from the voxel to the camera optical center is calculated. If the absolute value of the difference between the distance and the depth value is less than a preset tolerance threshold, the visibility coefficient value is determined to indicate that the image is not occluded; otherwise, the visibility coefficient value is determined to indicate that the image is occluded.

4. The method according to claim 1, characterized in that, The visibility coefficient is used to perform weighted back projection and spatial fusion on the two-dimensional semantic credibility score map to generate a three-dimensional language evidence body, including: For each voxel, the visibility of the voxel under multiple viewing angles is determined based on the visibility coefficient, and the target viewing angle of the voxel is selected. The semantic fusion score of the voxel is obtained by weighted averaging the scores of the corresponding pixel positions in the two-dimensional semantic credibility score map corresponding to the target viewpoint. The three-dimensional language evidence body is generated based on the semantic fusion scores of multiple voxels; the three-dimensional language evidence body is used to characterize the degree of semantic matching between each location in three-dimensional space and the natural language text query command.

5. The method according to claim 1, characterized in that, From the 3D scene data of the region to be located, obtain RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the region to be located, including: The environmental data of the area to be located is acquired by an RGB-D camera, and the three-dimensional scene data is generated based on the environmental data. Based on a preset sampling strategy, RGB image sequences and corresponding depth map sequences covering multiple viewpoints of the region to be located are filtered from consecutive frames of the 3D scene data; wherein, the sampling strategy includes filtering based on pose changes and image sharpness changes between adjacent viewpoints.

6. The method according to claim 1, characterized in that, Extract the text embedding vector of the natural language text query command and the visual feature maps of the RGB images in the RGB image sequence from multiple perspectives, including: Using a pre-trained two-dimensional visual language model, the text embedding vector of the natural language text query instruction and the visual feature maps of the RGB images in the RGB image sequence from multiple perspectives are extracted respectively; wherein, the pre-trained two-dimensional visual language model adopts the CLIP model or the GLIP model.

7. The method according to claim 1, characterized in that, Analyzing the three-dimensional language evidence and outputting the three-dimensional coordinates of the target object in the region to be located includes: The three-dimensional language evidence body is filtered using an adaptive threshold to obtain the filtered voxel region; Three-dimensional connected component clustering is performed on the filtered voxel region to extract multiple voxel clusters, and the target object is determined from the multiple voxel clusters. Calculate the geometric center of the voxel clusters of the target object to generate the three-dimensional coordinates of the target object.

8. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, perform the method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, perform the method as described in any one of claims 1 to 7.

Images