Target level three-dimensional point cloud cross-modal semantic retrieval method and device and electronic equipment

Abstract

The application discloses a target-level three-dimensional point cloud cross-modal semantic retrieval method and device and electronic equipment, comprising: segmenting an independent three-dimensional target point cloud from a three-dimensional scene point cloud and giving a unique identifier; projecting each target point cloud to three orthogonal two-dimensional observation planes to generate a multi-view synthesis image; using a pre-trained visual-language basic model to encode and fuse the synthesis image to generate a unified multi-modal feature vector; and constructing a vector database associated with the feature vector and its identifier. The natural language query text is encoded into a text query vector using the model, the semantic similarity between the text query vector and the feature vector in the database is calculated, and the corresponding three-dimensional target point cloud is located and returned. The application realizes accurate and efficient cross-modal retrieval from natural language to three-dimensional point cloud targets, solves the problem that traditional methods are difficult to support semantic fine-grained retrieval of three-dimensional target point clouds, and provides key technical support for training data management in fields such as embodied intelligence.

Description

Target-level 3D point cloud cross-modal semantic retrieval method, device, and electronic equipment Technical Field

[0001] This application relates to the fields of 3D computer vision and cross-modal retrieval technology, and in particular to a target-level 3D point cloud cross-modal semantic retrieval method, apparatus, and electronic device. Background Technology

[0002] With the widespread adoption of 3D sensing technologies such as LiDAR, the acquisition of massive amounts of 3D point cloud data has become a reality. How to efficiently and accurately retrieve specific targets of interest to users from this unordered and unstructured point cloud data is a significant challenge in the field of 3D vision. Existing 3D point cloud retrieval methods mainly focus on matching the similarity between point clouds, which has significant limitations in terms of flexibility and depth of understanding. On the one hand, they struggle to directly support users in querying using natural language, the most intuitive method; on the other hand, they typically lack the ability to represent and retrieve independent, complete individual targets within a scene, failing to meet the urgent need for fine-grained target-level retrieval, such as for specific vehicle models in autonomous driving or specific tools in robotic operations.

[0003] Furthermore, existing methods suffer from gaps in the construction of cross-modal retrieval frameworks: most studies focus only on improving single aspects such as point cloud feature extraction or similarity measurement, failing to systematically integrate and optimize key technologies such as accurate segmentation of target point clouds, robust multi-view feature learning, deep semantic alignment, and efficient index retrieval. This leads to existing solutions often facing problems such as large semantic gaps, limited retrieval accuracy, and weak system generalization capabilities in practical applications, making it difficult to meet the urgent need for flexible, accurate, and semantic retrieval of specific 3D targets in real-world scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, and electronic device for cross-modal semantic retrieval of target-level 3D point clouds, so as to solve the problem that traditional methods are difficult to support semantic fine-grained retrieval of 3D target point clouds.

[0005] According to a first aspect of the embodiments of this application, a method for cross-modal semantic retrieval of target-level 3D point clouds is provided, comprising:

[0006] Extract and segment independent 3D target point clouds from the 3D scene point cloud, and generate a unique identifier for each 3D target point cloud;

[0007] Each of the three-dimensional target point clouds is projected onto three orthogonal two-dimensional observation planes to generate a multi-view composite image containing the target's front, side, and top views;

[0008] The multi-view synthesized image is encoded and fused using a pre-trained vision-language basic model to generate a unified multimodal feature vector;

[0009] All the multimodal feature vectors and their identifiers are stored as associated data pairs in a vector database, and all 3D target point clouds are stored in a persistent file system;

[0010] The system accepts natural language query text input by the user and encodes the query text using the vision-language base model to obtain a text query vector.

[0011] Calculate the semantic similarity between the text query vector and the target multimodal feature vectors in the vector database, filter out feature vectors with similarity higher than a preset threshold, and locate and return the corresponding three-dimensional target point cloud from the persistent file system based on the identifier associated with the filtered feature vectors, as the final semantic retrieval result.

[0012] According to a second aspect of the embodiments of this application, a target-level three-dimensional point cloud cross-modal semantic retrieval device is provided, comprising:

[0013] The point cloud target aggregation module is used to extract and segment independent 3D target point clouds from the 3D scene point cloud, and generate a unique identifier for each 3D target point cloud;

[0014] The multi-view synthesis module is used to project each of the three-dimensional target point clouds onto three orthogonal two-dimensional observation planes to generate a multi-view synthesized image containing the target's front view, side view, and top view perspectives;

[0015] The multimodal encoding module is used to encode and fuse the multi-view synthetic image using a pre-trained vision-language basic model to generate a unified multimodal feature vector;

[0016] The vector library construction module is used to store all the multimodal feature vectors and their identifiers as associated data pairs in the vector database, and to store all 3D target point clouds in a persistent file system;

[0017] The query understanding module is used to accept natural language query text input by the user and encode the query text using the vision-language basic model to obtain a text query vector.

[0018] The cross-modal retrieval module is used to calculate the semantic similarity between the text query vector and the target multimodal feature vectors in the vector database, filter out feature vectors with similarity higher than a preset threshold, and locate and return the corresponding original 3D target point cloud from the persistent file system based on the unique identifier associated with the filtered feature vector, as the final semantic retrieval result.

[0019] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0020] One or more processors;

[0021] Memory, used to store one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0023] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0024] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0025] This application segments independent target point clouds from a 3D scene, orthogonally projects them onto three views to generate a synthetic image, and then uses a pre-trained vision-language basic model to encode and fuse multi-view information into a shared semantic space aligned with text, ultimately constructing a unified target-level cross-modal retrieval library. This approach effectively overcomes the limitations of traditional methods, such as difficulty in directly understanding natural language queries and insufficient utilization of the global geometric and semantic information of 3D targets, achieving accurate and efficient cross-modal retrieval from natural language to 3D point cloud targets. This method significantly improves the accuracy and efficiency of target-level semantic retrieval in complex scenes, providing reliable technical support for the intelligent retrieval and management of specific 3D targets in practical applications such as autonomous driving and robot interaction. Attached Figure Description

[0026] Figure 1 is a flowchart illustrating a target-level three-dimensional point cloud cross-modal semantic retrieval method according to an exemplary embodiment.

[0027] Figure 2 is a diagram illustrating the effect of search results according to an exemplary embodiment.

[0028] Figure 3 illustrates a target-level 3D point cloud cross-modal semantic retrieval device according to an exemplary embodiment.

[0029] Figure 4 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0030] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0031] Figure 1 is a flowchart illustrating a target-level 3D point cloud cross-modal semantic retrieval method according to an exemplary embodiment. Referring to Figure 1, a target-level 3D point cloud cross-modal semantic retrieval method provided by an embodiment of the present invention may include:

[0032] S1: Extract and segment independent 3D target point clouds from the 3D scene point cloud, and generate a unique identifier for each 3D target point cloud;

[0033] Specifically, in the field of embodied intelligence, especially in the application scenarios of indoor service robots, the target scene is typically a global 3D point cloud collected by the robot as it explores indoor environments such as homes, warehouses, or offices using its onboard depth camera or LiDAR. This point cloud data densely records various manipulable target objects (such as mugs on tables, books on bookshelves, etc.) and their spatial layout. This step aims to accurately separate and reconstruct each independent 3D target instance point cloud with complete physical dimensions and geometry from this dynamic, complex, and often occluded indoor scene stream data, and assign it a globally unique identifier.

[0034] This step includes the following sub-steps:

[0035] S11: Acquire single-frame point clouds and poses collected by sensors at multiple consecutive moments, and fuse them to obtain a 3D scene point cloud;

[0036] Specifically, it involves acquiring single-frame point cloud datasets collected by sensors such as lidar at multiple consecutive time points (T1, T2, ..., Tn). and its corresponding sensor precise pose set By using point cloud registration and fusion algorithms (such as ICP or odometry-based fusion), the single-frame point cloud dataset is... According to its corresponding pose The points are uniformly transformed to the same global world coordinate system, forming a complete and dense 3D scene point cloud S.

[0037] S12: Perform 3D bounding box annotation on the target object in each frame of point cloud, and obtain the transformation matrix of the corresponding local point cloud subset and bounding box;

[0038] Specifically, for each single frame of point cloud Perform 3D bounding box annotation to obtain the 3D bounding box. Ensure the 3D bounding box The position, size, and orientation of the bounding box are aligned with the target object. From its center point coordinates (i.e., the x, y, z coordinates of the center point), dimensions (i.e., length, width, and height) and rotation angle (i.e., roll angle, pitch angle, yaw angle) Parametric definition. Based on the bounding box. The parameters, from the original frame point cloud All points completely within the bounding box are cropped out to form a subset of the local point cloud of the target in the current frame. Define the transformation matrix based on the center and orientation of the bounding box. The transformation matrix This describes how to transform a bounding box to the actual detected bounding box. The position and orientation are key parameters for coordinate unification during subsequent cross-frame aggregation.

[0039] S13: For multiple 3D bounding boxes of the same target in different frames, the local point cloud subsets corresponding to each frame are uniformly transformed to the global coordinate system using the sensor pose, and the local point cloud subsets are aggregated to the local coordinate system of the first frame through the transformation matrix of each bounding box to obtain a complete and dense 3D target point cloud.

[0040] Specifically, for the same physical target in the scene at different times Multiple bounding boxes (i=1,2,…) and its corresponding local point cloud subset A two-stage coordinate transformation and aggregation process is performed.

[0041] Phase 1 (Global Alignment): For each time step Using the sensor pose transformation matrix at that moment Subset of local point cloud Transforming from its sensor coordinate system to the global world coordinate system, we obtain .

[0042] The second stage (local aggregation): To construct a dense target point cloud with consistent pose centered on the target itself, the moment when the target was first labeled is selected (e.g., the moment when the target is first labeled). bounding box Transformation matrix As a reference frame. For point clouds that have been transformed to the global coordinate system. It needs to be further transformed to this reference frame.

[0043] That is, for any point in In the reference frame The transformation result below It is obtained in the following way:

[0044]

[0045] in and They are time points and reference frame time The pose transformation matrix, For a moment The transformation matrix of the labeled bounding box. As a frame of reference, and This is the inverse of the corresponding matrix.

[0046] This step is equivalent to first transforming the point cloud to an intermediate state through global pose transformation, and then uniformly aligning it to the bounding box coordinate system of the reference frame using a bounding box matrix. Finally, all point clouds aligned to this reference frame coordinate system are... By aggregating the data, a complete and dense three-dimensional point cloud of the target is obtained. .

[0047] S14: Randomly generate a universally unique identification code for the three-dimensional target point cloud as its globally unique identifier;

[0048] Specifically, for each aggregated, generated, independent 3D target point cloud A globally unique identifier is generated randomly using a Universally Unique Identifier (UUID) algorithm. .Should Will be with point clouds Permanent binding serves as the core key for indexing, associating, and retrieving in all subsequent processes.

[0049] This completes the decomposition and identification of the scene into independent target instances, laying the foundation for subsequent multi-view generation and cross-modal coding.

[0050] This step addresses the core challenges of embodied intelligent indoor scenarios, providing a systematic solution. First, through multi-frame fusion reconstruction, it fundamentally solves the problem of incomplete single-view perception, generating high-quality, complete 3D models for each target that can be accurately identified. This far surpasses the reliability of retrieval based on incomplete cloud data from a single frame. Second, by assigning a globally unique ID to each instance, it achieves a leap from scene-level perception to target-level entity management. This provides a precise data foundation for subsequent fine-grained target retrieval based on natural language (such as "find the blue-covered book I just put on the sofa"), and is also a key prerequisite for constructing instance-level, multimodal datasets suitable for training robot operational skills.

[0051] S2: Project each of the three-dimensional target point clouds onto three orthogonal two-dimensional observation planes to generate a multi-view composite image containing the target's front, side, and top views;

[0052] Step S1 yields an independent and complete 3D target point cloud. After obtaining the precise 3D bounding box, this step aims to convert the unstructured 3D point cloud into a 2D multi-view representation that fully preserves its key appearance and structural information and conforms to the standard visual model input format. This lays the foundation for subsequent semantic encoding using a vision-language foundation model. This step includes the following sub-steps:

[0053] S21: Establish a local coordinate system for each target point cloud with the center of its 3D bounding box as the origin of the coordinate system and the front, side and top as the main axes.

[0054] Specifically, for each three-dimensional target point cloud A standardized local coordinate system for the target is established using the 3D bounding box described in S12. Let the center coordinates of this bounding box be (...). , , The unit vectors of its three principal axes (usually corresponding to the length, width, and height of the target) are respectively , , .by( , , () as the origin, with For the X-axis (front). Y-axis (lateral). Establish a right-handed Cartesian coordinate system with the Z-axis (above). This step directly reuses precise annotation information, ensuring that the projected views of different targets and placement postures have a consistent and semantically clear perspective (front / side / top view), which is key to view standardization.

[0055] S22: In the target local coordinate system, the target point cloud is projected onto the corresponding orthogonal planes along the three principal axes, and the pixel density of each projection plane is dynamically adjusted according to the target size through an adaptive resolution mechanism.

[0056] Specifically, the target point cloud is orthogonally projected onto the corresponding YZ plane (front view), XZ plane (side view), and XY plane (top view) along the X-axis (front-back), Y-axis (left-right), and Z-axis (top-bottom) directions. Considering the significant size variations of indoor objects (from small remote controls to large sofas), an adaptive resolution mechanism is employed: pixel density is dynamically calculated based on the physical dimensions of the target's bounding box in each projection direction. Let the dimensions (length, width, height) of the bounding box along the three principal axes be (L, W, H), and the preset baseline resolution be... The resolution of the three views can then be dynamically calculated as follows:

[0057] Front view resolution:

[0058] Side view resolution:

[0059] Top view resolution:

[0060] in This is the floor function. During projection, for a point in the point cloud... (Local coordinates), their pixel coordinates on the front view. The calculation is as follows:

[0061]

[0062]

[0063] Here, `floor` is the floor function, and `W` and `H` are the dimensions of the bounding box in the Y and Z directions, respectively. The pixel coordinate calculations for the side and top views are similar. This mechanism ensures that, regardless of object size, its projected image captures surface details with appropriate sharpness.

[0064] S23: For each projection plane after adjusting pixel density, fuse the visible surface and reflection intensity information of the point cloud into a three-channel color view;

[0065] Specifically, for each pixel unit on each projection plane, information fusion rendering is performed to generate a color view that is closer to a natural image and richer in information. All points projected onto that pixel unit are collected, and the following information is fused:

[0066] Surface geometric information: Calculate the average normal vector n of the point projected onto the pixel unit, and multiply it by the preset virtual light source direction l to simulate Lambertian illumination, thereby obtaining the geometric brightness value. .

[0067] Reflection intensity information: Calculate the average reflection intensity value of the point projected onto the pixel unit. .Will and After normalization, the colors are then mapped using a predefined color mapping function. The fused encoding is converted into a single three-channel RGB value:

[0068]

[0069] in and These are adjustable weighting coefficients. For pixels to which no point is projected, their RGB values ​​are set to (0, 0, 0) (black). Ultimately, each view generates a three-channel color image.

[0070] S24: Arrange the three-channel color views according to a predefined spatial layout, and perform style normalization and feature alignment through a lightweight convolutional network to finally generate a structurally consistent multi-view composite image containing front, side, and top views of the target.

[0071] Specifically, the generated front-view, side-view, and top-view color views are stitched together in the spatial arrangement of front-view, side-view, and top-view to form a single composite image. Then, a lightweight convolutional network is used to perform style normalization and cross-view feature alignment on this composite image. This convolutional network is trained on a large number of multi-view images to learn to eliminate style differences caused by acquisition conditions (such as lighting and sensor noise) and implicitly align the latent feature representations of the three sub-views, ultimately outputting a standardized and structured multi-view composite image. .

[0072] This step is the core conversion link connecting 3D geometric perception and 2D semantic understanding. By reusing precise bounding boxes to construct standard views and compressing 3D information into 2D multi-view images, this method can directly utilize visual-language models pre-trained on massive image-text pairs with strong generalization capabilities. This avoids the need to customize complex dedicated networks for 3D point clouds, significantly reducing the implementation threshold and enhancing the potential for semantic understanding.

[0073] S3: The multi-view synthesized image is encoded and fused using a pre-trained vision-language basic model to generate a unified multimodal feature vector;

[0074] To achieve a closed loop of "understanding-finding", the target multi-view composite image obtained through step S2 is used. This process transforms the images into a mathematical representation that can be processed by a machine and is aligned with the semantics of human language. This step utilizes a vision-language foundational model pre-trained on massive amounts of internet image-text pair data to encode and fuse multi-view images into a high-dimensional feature vector containing rich semantic information and existing in a semantic space shared with the text, namely the "multimodal feature vector".

[0075] The vision-language foundational model includes: a visual encoder, a text encoder, and a cross-modal alignment and fusion module, wherein:

[0076] The visual encoder consists of a pre-trained deep visual neural network, which receives the multi-view synthesized image, extracts deep visual features of the image through multi-layer nonlinear transformation, and finally outputs a high-dimensional visual feature vector.

[0077] The text encoder consists of a pre-trained deep language neural network, which receives the natural language query text, extracts deep semantic features of the text through semantic understanding, and finally outputs a high-dimensional text feature vector.

[0078] The cross-modal alignment and fusion module is used to constrain and optimize the outputs of the visual encoder and the text encoder during the model pre-training stage, so that the visual feature vector and the text feature vector are embedded in the same shared semantic feature space, thereby realizing a unified representation and measurement of image content and text semantics. During the model application stage, the module directly outputs the high-dimensional visual feature vector extracted by the visual encoding branch as the unified multimodal feature vector.

[0079] Step S3 includes the following sub-steps:

[0080] S31: The multi-view synthesized image is input into the visual encoder of the visual-language basic model for encoding. Specifically, the deep visual features of the image are extracted through the multi-layer nonlinear transformation of the visual encoder to form a unified visual feature vector.

[0081] Specifically, the standardized multi-view composite image generated in step S2 The input is fed into the visual encoder of the pre-trained vision-language base model. The visual encoder is a deep convolutional neural network.

[0082] During the forward propagation of the model First, the image is decomposed into several image patches and converted into an embedding sequence, which is then subjected to a series of multi-layer nonlinear transformations by the encoder (including multiple self-attention layers and a feedforward network). Because It is a single image stitched together from three views: front, side, and top. In the deep network of the encoder, these local features from different perspectives interact and merge naturally in their latent representation space as the receptive field expands and the features become more abstract, ultimately outputting a high-dimensional tensor. This tensor is further aggregated into a fixed-dimensional unified visual feature vector. .

[0083] S32: Using the cross-modal alignment and fusion module of the vision-language basic model, the high-dimensional visual feature vector is directly output as the unified multimodal feature vector.

[0084] Specifically, the visual feature vector The cross-modal alignment and fusion module is input into the vision-language foundation model. The core functionality of this module was achieved during the model's pre-training phase through objectives such as contrastive learning: that is, within the shared semantic embedding space, it brings matching image-text feature pairs closer together and pushes away mismatched feature pairs. Therefore, at this application stage... It is already located in a common high-dimensional feature space aligned with the text semantics. The cross-modal alignment fusion module contains a lightweight projection layer, which functions to... The final mapping is a unified multimodal feature vector that is completely consistent with the text feature vector format. .

[0085] This step is crucial for endowing the system with semantic understanding capabilities. By directly utilizing a visual-language model pre-trained on massive amounts of internet data and possessing strong generalization capabilities, this method avoids the massive labeled data and enormous computational overhead required to train a dedicated cross-modal model from scratch for indoor point cloud targets. This achieves efficient zero-shot semantic capability transfer, greatly improving the practicality and scalability of the solution. More importantly, the generated multimodal feature vectors... Built into a text-aligned semantic space, it fundamentally bridges the "semantic gap" between visual representation and linguistic description, enabling accurate cross-modal retrieval with only simple vector similarity calculations. This highly compact and semantically rich vector is not only well-suited for building efficient vector indexes to support real-time retrieval, but also robustly represents the target's category, shape, pose, and even some material properties. This allows it to accurately respond to complex, fine-grained natural language commands such as "search for cylindrical metal cans" or "vertically placed books," laying a solid technical foundation for building an efficient, accurate, and practical target-level 3D point cloud semantic retrieval system.

[0086] S4: Store all the multimodal feature vectors and their identifiers as associated data pairs in a vector database, and store all 3D target point clouds in a persistent file system; this step aims to process the multimodal feature vectors obtained in the previous steps. Organize and store the original 3D target point cloud data to build a digital asset library that supports efficient cross-modal retrieval.

[0087] This step includes the following sub-steps:

[0088] S41: For the unified multimodal feature vector generated for each target in step S3 and the globally unique identifier assigned in step S1 The two are encapsulated into an inseparable set of related data pairs. . As a computable representation in a high-dimensional semantic space It serves as the sole pointer to the target's original geometric data.

[0089] S42: Put all related data pairs Batch import a dedicated vector database. This database is deeply optimized for similarity searching of high-dimensional vectors. During the import process, the database uses the Navigable Small World Graph (HNSW) indexing algorithm based on the vector distribution to construct an efficient index structure in memory or on disk. This index enables subsequent calculations of text query vectors and massive multimodal feature vectors. When calculating the cosine similarity between two objects, a time-consuming full scan is performed out of order, but the most similar result can be returned in a very short time using an approximate nearest neighbor search.

[0090] S43: Combine all independent 3D target point clouds Its corresponding As another set of associated data, it is stored in a persistent file system. For ease of management, a directory structure organized by category is used for storage. Each Point clouds are typically saved in the common point cloud file format pcd to ensure that their complete geometric and attribute information is preserved.

[0091] S44: Create and maintain a bidirectional indexed mapping table. The core of this table is... It is logically strictly associated with a single record in the vector database (storage). ) and a physical file in the file system (storage) This mapping ensures that the results of semantic retrieval can be parsed unambiguously and with low latency, and located to the corresponding original 3D target point cloud data.

[0092] This step completes a crucial transformation from data processing to service-oriented retrieval, with its systematic storage architecture delivering significant performance and practical advantages. By separating and storing lightweight semantic feature vectors from the massive raw point cloud, and leveraging the strengths of vector databases and file systems respectively, this method ensures both retrieval speed and the secure and efficient management of raw data. The introduction of a dedicated vector database and its efficient indexes enables the system to filter candidates most similar to natural language descriptions from massive targets within milliseconds. Simultaneously, a strong association mapping established through unique identifiers ensures accurate and reliable backtracking from semantic matching results to raw 3D geometric data, providing a complete training loop for target-specific training. This architecture not only improves retrieval efficiency but also gives the entire system excellent scalability, allowing for dynamic addition and deletion of the target library as the robot's exploration environment expands, providing robust data support for robot algorithms requiring long-term fine-tuning.

[0093] S5: Accept the natural language query text input by the user and encode the query text using the aforementioned visual-language foundational model to obtain a text query vector. This step provides an interface for efficient and semantic queries on the constructed 3D target point cloud training database. In embodied intelligence model training and evaluation scenarios, researchers or system administrators often need to flexibly retrieve subsets that meet specific semantic conditions from a massive target instance library for purposes such as "retrieving all chair point clouds of different colors for data augmentation" or "finding all 'graspable' small tools for skill training." This step aims to transform such unstructured natural language queries describing training data requirements into mathematical vectors that machines can accurately calculate and that are aligned with stored visual features in a unified semantic space—that is, text query vectors.

[0094] This step includes the following sub-steps:

[0095] S51: The syntax-aware query parsing module decomposes the natural language query input by the user into entity words, attribute descriptions and relational phrases;

[0096] Specifically, the input natural language query statement (For example: "a cylindrical object with a spiral texture on its surface") is first fed into a syntax-aware query parsing module. This module combines dependency parsing and domain terminology recognition technology to decompose the query into key semantic components: target entity category or concept (such as "object") and attribute description (such as "cylindrical" and "spiral texture").

[0097] S52: Utilize a pre-constructed three-dimensional target knowledge graph to expand and disambiguate the parsed entity words, attribute descriptions, and relational phrases, generating a set of structured query representations rich in three-dimensional semantics;

[0098] Specifically, the parsed components are input into a pre-built, model-training-oriented 3D target knowledge graph for semantic standardization and expansion. This knowledge graph includes: (1) an object attribute system, such as geometric shape (cylindrical, cube), size category (small, large), surface characteristics (rough, smooth), material (metal, plastic), etc.; (2) functional or task tags, such as "stackable" and "gripable"; (3) synonyms and concept associations (e.g., "object" can be associated with all instances, "grasp" is associated with the "gripability" attribute). The system uses the graph to confirm the geometric concept of "cylindrical", standardize the surface attributes of "textured", and map "for grasping training" to the relevant functional attribute tags. Finally, a set of semantically rich structured query representations is generated. It integrates standardized concepts, attribute constraints, and functional labels.

[0099] S53: Input the structured query representation into the text encoder of the visual-language base model, and aggregate it through the query-conditional attention pooling layer to obtain the text query vector;

[0100] Specifically, the semantically enhanced structured query representation The input (constructed as a descriptive text) is fed into a text encoder, which is paired with the visual encoder in step S3 and belongs to the same pre-trained vision-language base model. The text encoder performs deep encoding on the input sequence, outputting a contextual feature vector for each semantic unit. To aggregate these features, this method employs a query-conditional attention pooling layer. This layer automatically evaluates and weights the features through a self-attention mechanism. The importance of different components (such as core categories, key attributes, and functional requirements) to the current query is determined, ultimately generating a fixed-dimensional, highly discriminative text query vector. . Multimodal feature vectors of all targets in the database They share the same semantic embedding space.

[0101] This step provides researchers and data management systems with a powerful and flexible natural language semantic retrieval interface, significantly improving the efficiency of managing and utilizing large-scale 3D target training datasets. Through deep parsing and domain knowledge graph enhancement, the system can accurately understand the complex filtering conditions of users for training data (such as "all objects with flat tops used for stability testing"), going beyond simple keyword matching. This enables researchers to quickly and accurately locate target point subsets that meet specific experimental needs or have specific attributes, efficiently constructing training or evaluation sets. The query-conditional attention mechanism ensures that the generated query vector highlights the core semantics of the query, thereby achieving high-precision retrieval in subsequent cross-modal matching. This capability makes this method not only a point cloud repository but also an intelligent training data management platform, directly supporting data-driven iterative processes in the development of embodied intelligence and other models, improving the efficiency of the entire development cycle.

[0102] S6: Calculate the semantic similarity between the text query vector and the target multimodal feature vectors in the vector database, filter out feature vectors with similarity higher than a preset threshold, and locate and return the corresponding 3D target point cloud from the persistent file system based on the identifier associated with the filtered feature vector, as the final semantic retrieval result. This step is the core operation for performing cross-modal semantic retrieval, aiming to transform the semantic query intent of the user (researcher or system) into specific 3D geometric instances returned from the target database. For embodied intelligence training scenarios, this process not only requires rapid initial screening of candidates from massive amounts of data, but also requires fine calibration of the candidate results to return a highly relevant and high-quality subset of target points, meeting the stringent requirements for constructing a specific training or test set.

[0103] S61: Perform an approximate nearest neighbor search in the vector database to obtain the top N multimodal feature vectors with the highest similarity to the text query vector;

[0104] Specifically, the text query vector obtained in step S5 As query input, an approximate nearest neighbor (ANN) search is performed on the vector database built by S4. The vector database utilizes its pre-built, efficient indexes (such as the HNSW graph) for fast computation. With all multimodal feature vectors in the library Calculate the cosine similarity between the vectors and return the top N candidate vectors with the highest similarity and their corresponding identifiers. This step is completed in milliseconds, enabling rapid initial screening of a large-scale target library.

[0105] S62: For the top N candidate feature vectors, a lightweight reordering network is used for fine matching. This network integrates a cross-modal interactive attention mechanism and a local feature alignment module, and outputs a calibrated semantic similarity score.

[0106] Specifically, to improve retrieval accuracy, the top N coarsely screened candidates are finely reordered. The text query vector... With candidate visual feature vectors The input is a lightweight reordering network. This network is a shallow cross-modal Transformer module, with a cross-modal interaction attention mechanism at its core. As a query As key / value pairs, attention calculations allow text features to actively "examine" and focus on the most relevant parts of each visual feature. Simultaneously, the local feature alignment module within the network further performs fine-grained comparisons of the aligned cross-modal features. The re-ranking network outputs a set of semantic similarity scores calibrated through deep interaction. This score is a better indicator of the complexity of semantic matching than the initial cosine similarity.

[0107] S63: Determine the final set of positive matching targets based on the calibrated semantic similarity score and the threshold dynamically adjusted according to the application scenario;

[0108] Specifically, based on the calibrated score Determine the final positive match target. The system sets a dynamic threshold strategy: this threshold can be adjusted according to the specific application scenario (such as data augmentation tasks requiring high precision or high recall). The score is higher than the currently set threshold The candidate targets were determined as the final positive matching targets, forming a set. This step ensures the flexibility and usability of the search results.

[0109] S64: Retrieve the corresponding 3D point cloud data from the persistent file system by using the identifier bound to the multimodal feature vector of the positive matching target;

[0110] Specifically, as shown in Figure 2, based on the final matched set of target identifiers The system uses a bidirectional index mapping table established by S4 to accurately locate the corresponding original 3D point cloud file stored in the persistent file system. The result is then loaded and returned as the final result of this semantic search.

[0111] This step employs a two-stage retrieval architecture of "rapid initial screening" and "refined reordering," which significantly improves the accuracy and reliability of results while ensuring retrieval efficiency, perfectly meeting the high quality and accuracy requirements of embodied intelligence training data management. Efficient near-nearest neighbor search enables the system to handle the ever-growing database size, while the lightweight reordering network, through deep cross-modal understanding, effectively distinguishes between surface similarity and semantically true matching targets, such as differentiating "a circular mat for placement" from "a sphere for rolling." A dynamic threshold mechanism provides researchers with flexible result filtering control, allowing them to adjust the retrieval stringency according to different training tasks (such as precise imitation learning or diversified exploration). Finally, a unique identifier enables lossless and accurate backtracking from semantic matching to raw 3D geometric data, providing ready-to-use, reliable, high-quality data for subsequent simulation training, skill learning, or model evaluation, significantly improving the intelligence level and overall efficiency of data preparation and utilization in the embodied intelligence R&D process.

[0112] Figure 3 is a block diagram of a target-level 3D point cloud cross-modal semantic retrieval device according to an exemplary embodiment. Referring to Figure 3, the device includes:

[0113] Point cloud target aggregation module 1 is used to extract and segment independent 3D target point clouds from 3D scene point clouds, and generate a unique identifier for each 3D target point cloud;

[0114] The multi-view synthesis module 2 is used to project each of the three-dimensional target point clouds onto three orthogonal two-dimensional observation planes to generate a multi-view synthesized image containing the target's front view, side view, and top view perspectives;

[0115] Multimodal encoding module 3 is used to encode and fuse the multi-view synthesized image using a pre-trained vision-language basic model to generate a unified multimodal feature vector;

[0116] Vector library construction module 4 is used to store all the multimodal feature vectors and their identifiers as associated data pairs in the vector database, and store all three-dimensional target point clouds in a persistent file system;

[0117] The query understanding module 5 is used to accept natural language query text input by the user and encode the query text using the vision-language basic model to obtain a text query vector.

[0118] The cross-modal retrieval module 6 is used to calculate the semantic similarity between the text query vector and the target multimodal feature vectors in the vector database, filter out feature vectors with similarity higher than a preset threshold, and locate and return the corresponding original three-dimensional target point cloud from the persistent file system based on the unique identifier associated with the filtered feature vector, as the final semantic retrieval result.

[0119] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the target-level 3D point cloud cross-modal semantic retrieval method as described above. Figure 4 shows a hardware structure diagram of any data processing capability device in which the target-level 3D point cloud cross-modal semantic retrieval device provided in this embodiment of the invention is located. Besides the processor and memory shown in Figure 4, the data processing capability device in this embodiment may also include other hardware depending on the actual function of the data processing capability device, which will not be elaborated further.

[0120] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the target-level 3D point cloud cross-modal semantic retrieval method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of a wind turbine, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0121] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0122] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A target-level 3D point cloud cross-modal semantic retrieval method, characterized in that, include: Extract and segment independent 3D target point clouds from the 3D scene point cloud, and generate a unique identifier for each 3D target point cloud; Each 3D target point cloud is projected onto three orthogonal 2D observation planes to generate a multi-view composite image containing front, side, and top views of the target. A pre-trained visual-language model is used to encode and fuse the multi-view composite image, generating a unified multimodal feature vector. All multimodal feature vectors and their identifiers are stored as associated data pairs in a vector database, and all 3D target point clouds are stored in a persistent file system. Natural language query text input by the user is accepted, and the query text is encoded using the visual-language model to obtain a text query vector. The semantic similarity between the text query vector and each target multimodal feature vector in the vector database is calculated. Feature vectors with similarity higher than a preset threshold are selected. Based on the identifier associated with the selected feature vector, the corresponding 3D target point cloud is located in the persistent file system and returned as the final semantic retrieval result.

2. The method according to claim 1, characterized in that, The process involves extracting and segmenting independent 3D target point clouds from a 3D scene point cloud, and generating a unique identifier for each 3D target point cloud. This includes: acquiring single-frame point clouds and poses collected by sensors at multiple consecutive time points, and fusing them to obtain a 3D scene point cloud; annotating the target objects in each frame point cloud with 3D bounding boxes, and obtaining the transformation matrix of the corresponding local point cloud subset and the bounding box; for multiple 3D bounding boxes of the same target in different frames, using the transformation matrix of the sensor pose and the bounding box, uniformly transforming the corresponding local point cloud subsets of each frame to the same global coordinate system, and aggregating the local point cloud subsets to the local coordinate system of the first frame through the transformation matrix of each bounding box to obtain a complete and dense 3D target point cloud; and randomly generating a universally unique identification code for each 3D target point cloud as its globally unique identifier.

3. The method according to claim 1, characterized in that, Each of the three-dimensional target point clouds is projected onto three orthogonal two-dimensional observation planes to generate a multi-view composite image containing front, side, and top views of the target. This includes: establishing a local coordinate system for the target with the center of the three-dimensional bounding box of each three-dimensional target point cloud as the origin of the coordinate system, and front, side, and top as the main axes; in the local coordinate system, the target point cloud is projected onto the corresponding orthogonal planes along the three main axes, and the pixel density of each projection plane is dynamically adjusted according to the target size through an adaptive resolution mechanism; for each projection plane with adjusted pixel density, the visible surface and reflection intensity information of the point cloud are fused into a three-channel color view; the three-channel color views are arranged according to a predefined spatial layout, and style normalization and feature alignment are performed through a lightweight convolutional network, finally generating a structurally consistent multi-view composite image containing front, side, and top views of the target.

4. The method according to claim 1, characterized in that, The vision-language foundational model includes a visual encoder, a text encoder, and a cross-modal alignment and fusion module. The visual encoder, composed of a pre-trained deep visual neural network, receives the multi-view synthesized image and extracts deep visual features from the image through multi-layer nonlinear transformations, ultimately outputting a high-dimensional visual feature vector. The text encoder, composed of a pre-trained deep language neural network, receives the natural language query text and extracts deep semantic features from the text through semantic understanding, ultimately outputting a high-dimensional text feature vector. The cross-modal alignment and fusion module constrains and optimizes the outputs of the visual encoder and text encoder during the model pre-training stage, embedding the visual feature vector and text feature vector into the same shared semantic feature space, thereby achieving a unified representation and measurement of image content and text semantics. During the model application stage, this module directly outputs the high-dimensional visual feature vector extracted by the visual encoding branch as the unified multimodal feature vector.

5. The method according to claim 1, characterized in that, The multi-view synthesized image is encoded and fused using a pre-trained vision-language foundation model to generate a unified multimodal feature vector. This includes: inputting the multi-view synthesized image into the visual encoder of the vision-language foundation model for encoding, specifically by extracting deep visual features of the image through multi-layer nonlinear transformations of the visual encoder to form a high-dimensional visual feature vector; and using the cross-modal alignment and fusion module of the vision-language foundation model to directly output the high-dimensional visual feature vector as the unified multimodal feature vector.

6. The method according to claim 1, characterized in that, The system accepts natural language query text input by the user and encodes the query text using the visual-language foundation model to obtain a text query vector. This includes: decomposing the user-input natural language query into entity words, attribute descriptions, and relational phrases using a syntax-aware query parsing module; expanding and disambiguating the parsed entity words, attribute descriptions, and relational phrases using a pre-constructed 3D target knowledge graph to generate a set of structured query representations rich in 3D semantics; inputting the structured query representations into the text encoder of the visual-language foundation model and aggregating them through a query-conditional attention pooling layer to obtain the text query vector.

7. The method according to claim 1, characterized in that, The semantic similarity between the text query vector and the multimodal feature vectors of each target in the vector database is calculated. Feature vectors with similarity higher than a preset threshold are selected. Based on the identifiers associated with the selected feature vectors, the corresponding 3D target point clouds are located and returned from the persistent file system as the final semantic retrieval result. This includes: performing an approximate nearest neighbor search in the vector database to obtain the top N multimodal feature vectors with the highest similarity to the text query vector; performing fine matching on the top N candidate feature vectors using a lightweight reordering network that integrates a cross-modal interactive attention mechanism and a local feature alignment module, and outputting a calibrated semantic similarity score; determining the final positive matching target set based on the calibrated semantic similarity score and a threshold dynamically adjusted for the application scenario; and retrieving the corresponding 3D point cloud data from the persistent file system using the identifiers bound to the multimodal feature vectors of the positive matching targets.

8. A target-level 3D point cloud cross-modal semantic retrieval device, characterized in that, include: The point cloud target aggregation module is used to extract and segment independent 3D target point clouds from the 3D scene point cloud, and generate a unique identifier for each 3D target point cloud; The multi-view synthesis module is used to project each of the three-dimensional target point clouds onto three orthogonal two-dimensional observation planes to generate a multi-view synthesized image containing the target's front view, side view, and top view perspectives; A multimodal encoding module is used to encode and fuse the multi-view synthesized image using a pre-trained visual-language basic model to generate a unified multimodal feature vector. A vector library construction module is used to store all the multimodal feature vectors and their identifiers as associated data pairs in a vector database, and all 3D target point clouds are stored in a persistent file system. A query understanding module is used to accept natural language query text input by the user and encode the query text using the visual-language basic model to obtain a text query vector. A cross-modal retrieval module is used to calculate the semantic similarity between the text query vector and each target multimodal feature vector in the vector database, filter out feature vectors with similarity higher than a preset threshold, and locate and return the corresponding 3D target point cloud from the persistent file system based on the unique identifier associated with the filtered feature vector, as the final semantic retrieval result.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images