Text-to-point cloud cross-modal positioning method and system based on visual language model

Abstract

This invention belongs to the field of computer vision technology. It proposes a text-to-point cloud cross-modal localization method and system based on a visual language model. The method converts 3D point clouds into a bird's-eye view image and a structured scene graph, the latter containing nodes with semantic and spatial location information. Natural language descriptions, the bird's-eye view, and the scene graph are input into a pre-trained visual-language model to generate cross-modal alignment features. Through a partial node association mechanism, the semantic information is selectively mapped to spatially matching nodes to determine the target node. Then, based on its spatial location, the original point cloud is retrieved to achieve accurate localization. This invention utilizes dual structured representation and explicit semantic-spatial alignment to effectively bridge the semantic gap between point clouds and language, improving the accuracy, robustness, and interpretability of localization. It is applicable to scenarios requiring language-guided 3D perception, such as autonomous driving and robotics.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a text-to-point cloud cross-modal localization method and system based on a visual language model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence and multimodal perception technologies, cross-modal understanding and localization have become key capabilities for intelligent systems to achieve environmental cognition and human-computer interaction. In recent years, visual-language pre-trained models (such as CLIP, BLIP, and LLaVA) have demonstrated strong performance in semantic alignment tasks between images and text, driving significant progress in applications such as image-text retrieval, visual question answering, and instruction-based expression understanding. Meanwhile, 3D point clouds, as an important data form describing the geometric structure of the physical world, are widely used in fields such as autonomous driving, robot navigation, digital twins, and augmented reality. However, point cloud data is characterized by its disorder, sparsity, and unstructured nature, making it difficult to directly and efficiently align with natural language. To bridge this gap, researchers have begun exploring the transformation of point clouds into structured or image-based representations (such as voxel meshes, multi-view projections, and bird's-eye views) to adapt to the input paradigms of existing visual language models. Against this backdrop, how to construct a cross-modal framework that can effectively integrate 3D spatial semantics and natural language instructions to achieve accurate localization from text to point clouds has become an important technical direction connecting high-level semantic understanding and low-level geometric perception.

[0004] While existing methods have achieved good results in text localization tasks for 2D images, directly transferring them to 3D point cloud scenes still faces many challenges. First, point clouds lack a natural pixel structure, making it difficult to embed them directly into visual encoders like images, resulting in insufficient semantic information extraction. Second, natural language descriptions often involve relative positions, topological relationships, or functional attributes between objects, but traditional point cloud processing methods struggle to explicitly model these high-level semantic relationships, creating a semantic gap between language and geometry. Third, existing cross-modal models often employ full-image or global feature matching strategies, which are easily affected by irrelevant objects in complex 3D scenes, failing to achieve fine-grained, interpretable target localization. Furthermore, without structural abstraction of objects in the scene, the model struggles to establish a clear correspondence between entities mentioned in language and specific regions in the point cloud. Therefore, a mechanism is urgently needed that can both preserve the spatial structure of point clouds and deeply couple with language semantics, ensuring localization accuracy while improving the model's interpretability and generalization ability. Summary of the Invention

[0005] To address the problems in existing technologies that fail to effectively utilize the spatial information contained in language in complex 3D point cloud scenes and lack explicit spatial reasoning capabilities, this invention provides a text-to-point cloud cross-modal localization method and system based on a visual language model. By inputting a natural language description, the target location can be accurately inferred from a locally complex 3D point cloud map, greatly improving localization efficiency and accuracy.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a text-to-point cloud cross-modal localization method based on a visual language model.

[0007] A text-to-point cloud cross-modal localization method based on a visual language model includes the following steps: The 3D point cloud data to be located is converted into bird's-eye view imagery and scene graph structure. The scene graph structure contains multiple nodes, each of which is associated with semantic information and spatial location information in the bird's-eye view imagery. The user-input natural language description, bird's-eye view image, and scene graph structure are used as multimodal inputs and provided to a pre-trained large-scale vision-language model to obtain cross-modal features related to the natural language description through the cross-modal interaction mechanism of the large-scale vision-language model. Based on cross-modal features, a partial node association mechanism is adopted to selectively map the semantic information in the natural language description to at least some nodes in the scene graph structure in order to determine the target node consistent with the natural language description. Based on the spatial location information associated with the target node, determine the target location in the 3D point cloud data corresponding to the natural language description.

[0008] In one implementation of the first aspect of the present invention, converting the three-dimensional point cloud data to be located into a bird's-eye view image and a scene diagram structure includes: The points in the 3D point cloud data are projected onto the ground plane and then rasterized to generate a bird's-eye view image. Based on object instances in 3D point cloud data, a node is created for each object instance. The semantic information of the node is the semantic label of the object instance, and the spatial location information of the node is the pixel centroid coordinates of the object instance in the bird's-eye view image, thus forming a scene graph structure.

[0009] In one implementation of the first aspect of the present invention, a user-input natural language description, a bird's-eye view image, and a scene graph structure are provided as multimodal inputs to a pre-trained large-scale vision-language model, including: Visual features are obtained by processing bird's-eye view images using a visual encoder; Text features are obtained by processing natural language descriptions and scene graph structures through a text encoder; Visual and textual features are input into the text decoder of a large vision-language model to obtain cross-modal features.

[0010] As a further limitation of the first aspect of the present invention, text features are obtained by processing natural language descriptions and scene graph structures through a text encoder, including: Sequence the semantic and spatial location information of each node in the scene graph structure into a text sequence; The text sequence is concatenated with the natural language description to form a joint text input; Text features are obtained by encoding the joint text input using a text encoder.

[0011] In one implementation of the first aspect of the present invention, based on cross-modal features, a partial node association mechanism is employed to selectively map semantic information in the natural language description to at least a portion of the nodes in the scene graph structure, including: Based on cross-modal features, identify objects mentioned in natural language descriptions; For each mentioned object, calculate the distance between its theoretical spatial location in the natural language description and the spatial location information associated with each node in the scene graph structure; If the distance is less than a preset threshold, the mentioned object will be associated with the corresponding node and used as the target node.

[0012] In one implementation of the first aspect of the present invention, determining the target location in the three-dimensional point cloud data corresponding to the natural language description based on the spatial location information associated with the target node includes: Use the spatial location information associated with the target node as the two-dimensional pixel coordinates in the bird's-eye view image; The target position is obtained by back-projecting the two-dimensional pixel coordinates to the world coordinate system.

[0013] In one implementation of the first aspect of the present invention, before using the user-input natural language description, bird's-eye view image, and scene graph structure as multimodal input, the method further includes: Obtain the training dataset, which contains multiple samples. Each sample includes a segment of reference 3D point cloud data, a reference bird's-eye view image and a reference scene graph structure generated from the reference 3D point cloud data, and a reference natural language description generated for a specific location in the reference 3D point cloud data. Based on the training dataset, a large-scale vision-language model is fine-tuned and trained. The natural language description is automatically generated based on the relative spatial relationship of multiple object instances in a specific location and reference 3D point cloud data, the semantic category of the object instances, and the color attributes.

[0014] Secondly, the present invention provides a text-to-point cloud cross-modal localization system based on a visual language model.

[0015] A text-to-point cloud cross-modal localization system based on a visual language model includes: The point cloud conversion unit is configured to convert the 3D point cloud data to be located into a bird's-eye view image and a scene graph structure, wherein the scene graph structure contains multiple nodes, each node being associated with semantic information and spatial location information in the bird's-eye view image. The multimodal input unit is configured to take the user-input natural language description, bird's-eye view image, and scene graph structure as multimodal inputs and provide them to a pre-trained large-scale vision-language model to obtain cross-modal features related to the natural language description through the cross-modal interaction mechanism of the large-scale vision-language model. The node association unit is configured to: based on cross-modal features, adopt a partial node association mechanism to selectively map semantic information in the natural language description to at least a portion of the nodes in the scene graph structure, so as to determine the target node consistent with the natural language description; The location determination unit is configured to determine the target location in the 3D point cloud data corresponding to the natural language description based on the spatial location information associated with the target node.

[0016] Thirdly, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the text-to-point cloud cross-modal localization method based on a visual language model, which is the first aspect of the present invention.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the method for cross-modal localization from text to point cloud based on a visual language model, which is a first aspect of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a dual-structured representation of a bird's-eye view and a scene graph, fusing the geometric layout and semantic relationships in 3D point clouds into a visual-language model in a top-down manner. This allows the model to simultaneously acquire global spatial structure and local semantic details. Based on this, a partial node association mechanism guides the model to actively align textual cues with scene nodes, enabling explicit spatial logic in the cross-modal reasoning process, thereby achieving accurate understanding and localization inference of language descriptions. Through this process, this invention can fully exploit the inherent correlation between point cloud data and natural language in complex 3D environments, effectively improving the accuracy, robustness, and interpretability of localization. Specifically, this dual-structured representation, namely the bird's-eye view image converted from the original 3D point cloud and the scene graph structure containing semantic labels and pixel centroid coordinates, together with the user's input natural language description, serves as a multimodal input and is fed into a pre-trained large-scale vision-language model. This model, through its powerful cross-modal interaction mechanism, generates cross-modal features that are highly semantically aligned with the natural language description. Based on these features, a partial node association mechanism further selectively maps the semantic elements mentioned in the language to nodes with spatially matching locations in the scene graph, accurately identifying target nodes consistent with the text description, and finally, based on the spatial location information bound to the node, back-inferring the target region in the original point cloud. Thus, this invention achieves closed-loop alignment from unstructured point cloud to structured semantic graph and then to language instructions, not only overcoming the inherent defects of sparse and disordered point cloud data and difficulty in directly matching it with text, but also providing a reliable technical path for intelligent systems to understand and respond to natural language instructions in complex 3D environments.

[0019] In the point cloud preprocessing stage, this invention generates a bird's-eye view image by projecting the 3D point cloud onto a ground plane and rasterizing it. Simultaneously, a scene graph structure is constructed based on object instances in the point cloud, where each node not only carries the semantic label of the object but also accurately records its pixel centroid coordinates in the bird's-eye view. This dual representation strategy preserves the overall spatial layout and geometric distribution features of the scene (through the bird's-eye view) while explicitly modeling the semantic entities and their positional relationships within the scene (through the scene graph). Compared to methods using only the raw point cloud or a single view, this design provides richer and more structured input for subsequent vision-language models, enabling the model to simultaneously perceive global context and local semantic objects. This significantly enhances its ability to understand complex 3D scenes and lays a solid data foundation for subsequent accurate cross-modal alignment.

[0020] This invention abandons traditional full-graph matching or global attention mechanisms when determining target nodes, instead employing a partial node association mechanism based on cross-modal features. This mechanism first parses the specific objects mentioned in the natural language description, then calculates the geometric distance between the theoretical positions of these objects in the language space and the associated spatial positions of each node in the scene graph. Semantic mapping is only established when the distance is less than a preset threshold. This selective association strategy effectively avoids assigning semantic errors to spatially irrelevant interference nodes, especially in complex scenes containing many similar objects or with severe occlusion, significantly suppressing mismatches. Therefore, it not only improves the accuracy and robustness of target node recognition but also makes the localization results more granular and interpretable, ensuring that the final output position strictly corresponds to the actual physical region indicated by the language command.

[0021] This invention achieves effective fine-tuning of a large-scale vision-language model by constructing a training dataset specifically for 3D point cloud scenes. Each sample includes a reference point cloud, a bird's-eye view and scene graph generated from it, and a natural language description automatically generated based on spatial relationships, semantic categories, and attributes between objects. This training strategy enables the model to retain its powerful general cross-modal capabilities while deeply learning the inherent mapping rules between 3D spatial structure, object semantics, and natural language expressions. Compared to directly using an unadapted general model, the finely tuned model can more accurately understand language instructions involving complex spatial semantics such as relative position and topological relationships, significantly improving generalization ability and cross-modal alignment accuracy in real point cloud application scenarios, and solving the localization bias problem caused by the lack of 3D prior knowledge in general models.

[0022] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 A schematic diagram of a training process provided for an exemplary embodiment of the present invention; Figure 2 A schematic diagram of a partial node association strategy provided as an exemplary embodiment of the present invention; Figure 3 A schematic diagram comparing the positioning effect of the present invention with other methods, provided as an exemplary embodiment of the present invention; Figure 4 A schematic diagram of a text-to-point-cloud cross-modal localization system based on a visual language model provided as an exemplary embodiment of the present invention; Figure 5 A schematic diagram of a computer device provided for an exemplary embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] Text-to-point cloud localization aims to infer precise spatial location from a 3D point cloud map based on natural language descriptions, and is a key technology in scenarios such as embodied intelligence and autonomous driving. In typical applications, such as autonomous taxi services, the system often relies on the Global Navigation Satellite System (GNSS) for positioning. However, in complex urban environments, GNSS signals are often significantly degraded due to multipath effects and atmospheric delays, making it difficult to accurately determine the passenger's location. To compensate for this deficiency, users can describe their surroundings in natural language, allowing the system to extract additional spatial cues from the language and align them with the point cloud map, thereby achieving localization.

[0029] In recent years, various deep learning-based text-to-point cloud localization methods have emerged. The paper "Text2Pos: Text-to-point-cloud cross-modal localization," published at CVPR 2022, employs a two-stage strategy from coarse to fine: first, candidate sub-images are retrieved in a large-scale point cloud map, and then fine-tuning is performed within a smaller local area. Building on this paradigm, the paper "Text2Loc: 3D PointCloud Localization from Natural Language," published at CVPR 2024, uses a more advanced text encoder and introduces a Transformer-based cross-modal feature interaction mechanism to encode and aggregate text and point cloud instance features for end-to-end prediction, significantly improving retrieval and localization accuracy. However, existing methods still have shortcomings: First, subgraph retrieval and fine-grained localization are limited to a small and structurally simple subgraph range (30 meters on each side), which is significantly smaller than the actual visual observation range of humans and is difficult to cover highly complex urban environments; Second, existing methods generally use end-to-end regression to directly predict locations and lack a clear spatial relationship modeling mechanism, making it difficult for the model to perform spatial reasoning with interpretability and robustness.

[0030] Humans can naturally understand and describe spatial layouts spanning tens of meters in real-world environments, indicating that text-to-point cloud localization systems need cross-scale spatial understanding capabilities—that is, the ability to align a small amount of text description with complex point cloud scenes. In recent years, with the development of large-scale visual language models, based on their powerful multimodal understanding and reasoning capabilities, we can now handle complex spatial understanding tasks by linking linguistic descriptions with visual (or structured) input. Therefore, the introduction of large-scale visual language models offers new possibilities for achieving more refined, reliable, and interpretable text-to-point cloud localization.

[0031] In view of the problems existing in the current solution, this implementation proposes a text-to-point cloud localization method based on a large-scale vision-language model. First, a dataset containing a training set, a validation set, and tests in different scenarios is provided. The dataset contains two subsets: the first subset contains road point clouds collected by vehicle-mounted LiDAR, including instances of each point, semantic and color information, used for model training and testing; the second subset is aerial photogrammetric point clouds, containing the same point cloud format, used for cross-scene and cross-domain generalization testing. Location sampling and description generation are performed based on the point cloud information provided by the dataset. Each sample includes its corresponding map, spatial location, and text description. Then, the user inputs a natural language description, and the system converts the 3D point cloud map into two types of structured visual input: a bird's-eye view image, used to encode the overall geometric layout; and a scene graph, used to represent nodes composed of object semantic labels and center locations. The two structures described above together constitute the multi-layered contextual information of the point cloud, providing a foundation for subsequent multimodal reasoning. Next, the bird's-eye view imagery and scene graph structure are used as visual input, along with natural language descriptions, and fed into a pre-trained large-scale vision-language model. Through the model's internal attention mechanism and cross-modal interaction module, semantic cues contained in the language are extracted and aligned with the geometric and semantic structures of the map, thereby obtaining cross-modal spatial reasoning features. Finally, based on these cross-modal features, a partial node association mechanism is introduced. Based on the natural language description, nodes most relevant to the textual cues are selected and matched from the scene graph nodes. This mechanism can explicitly establish the correspondence between text and scene nodes, and infer the target location by combining point cloud geometric coordinates, ultimately outputting the precise spatial location in the 3D point cloud map.

[0032] More specifically, it includes the following processes: In the ground truth data generation stage, this invention first acquires a 3D point cloud map of the area to be located, including a series of examples. ,in Indicates the number of instances. Each object Include A point set consisting of points , including coordinates and the corresponding RGB information The point cloud data is then converted into two complementary representations: a bird's-eye view and a scene view.

[0033] For generating the bird's-eye view, the point cloud contained in the instance is projected onto the ground plane, and a representative color for each object instance is calculated, typically using the average color of the point cloud within the instance: (1); The projected point cloud was then rasterized to generate a size of [size missing]. For two-dimensional images, when pixels contain multiple categories, foreground objects are rendered first to preserve key features.

[0034] Simultaneously construct scene graph ,in Represents a set of nodes. Let the set of edges be denoted by , and to maintain computational efficiency, let . Each of the nodes This represents an object instance in the map, with attributes including the object's semantic label and the object's pixel centroid coordinates on the bird's-eye view: (2); This allows scene graphs to capture semantic information and relative spatial relationships between objects, providing a structured environmental description for subsequent processing.

[0035] This invention in text retrieval During the generation phase, a natural language description is automatically constructed based on the semantic category, spatial orientation, and color attributes of point cloud object instances. The rules for determining spatial orientation are as follows: (3); Let the query location be The point in the object instance that is closest to this location is Direction determination consists of two parts. First, it determines whether the position is on the object. If the Euclidean distance between the two is less than a preset threshold... (In this embodiment) If the query position is above the instance, then the query position is considered to be "above" the instance; otherwise, the relative displacement components between the query position and the center of the object instance in the horizontal plane are calculated. and And based on the above formula, calculate a direction belonging to east, south, west, or north. For color calculation, first, a series of color templates are predefined, including color names and their corresponding RGB values, and then the representative color of the calculation object instance is used. The distance between the standard RGB values ​​is used to select the closest color name as the color description for the object instance. Finally, the semantic, color, and direction attributes are combined to generate a text clause, with the following format: .

[0036] In this embodiment, each text retrieval The description consists of the six clauses of the text mentioned above.

[0037] The network model of this invention is a large-scale visual-language model based on an encoder-decoder architecture. The network structure can adopt existing visual-language models such as the Qianwen series or Shusheng series, which have the ability to align and generate images and text. The model mainly includes a visual encoder, a visual adapter, a text encoder, and a text decoder.

[0038] Bird's-eye view The data is divided into patches of a fixed size, let the patch size be... Then we get One patch. The visual encoder uses the Vision Transformer architecture to extract the high-dimensional features of the patch as a token, whose dimension is denoted as . The visual adapter is used to align the dimensions of visual features with those of text features, changing the image dimensions from... Transform into In addition, additional system instructions are defined. Instructions for tasks and input / output formats. The text encoder processes input system commands and text retrieval, first segmenting the input text data into words, dividing the original text into... Each token is a smallest semantic unit and is mapped to a unique index number in the vocabulary. The numerical ID is then queried in the vocabulary and mapped into a dense vector embedding, denoted by its dimension 1. Now, both the bird's-eye view and the text input are represented as a high-dimensional vector of several tokens, denoted as... The text decoder uses an autoregressive approach. Within each time step, a prediction vector is generated progressively, and the text token with the highest probability in the vocabulary is selected as the output. The cross-entropy loss function is used during model training. (4); The output at each time step depends on the historical output and multimodal input features.

[0039] During the model training and inference phases, the generated bird's-eye view is input into the visual encoder to extract visual features, while the scene map, system prompts, and natural language queries describing the target location are input into the text encoder to extract text features.

[0040] The training process is as follows Figure 1As shown, the process comprises four main parts: image input, text input, visual language model, and output. Image input is a bird's-eye view image of size H×W, used to represent the spatial layout of a 3D point cloud. Text input includes a scene graph (composed of multiple nodes representing semantic objects in the scene), system prompts, and user-input text. The system prompts define the model's role as a "bird's-eye view image localization expert assistant," while user input consists of natural language instructions describing the target location, such as "The target location is on the beige sidewalk, above the gray parking area." These multimodal inputs are fed into the visual language model, which uses LoRA technology for efficient parameter fine-tuning. The model output consists of two parts: first, a "partial node association" result, showing whether each semantic label (such as parking lot, building) is localizable and its matching node number; second, a "location estimation" result, outputting the target's 2D coordinates in the bird's-eye view image, such as [205, 148], achieving a precise mapping from natural language to specific spatial coordinates. The entire process embodies a complete closed loop from joint input of structured semantic graphs and images to cross-modal reasoning and localization output.

[0041] To address the issue of missing objects due to limited map coverage, this invention introduces a partial node allocation mechanism, such as... Figure 2 As shown, this mechanism explicitly supervises the model during decoding to determine whether the objects mentioned in the text query are visible within the current map area and establishes a correspondence between text objects and scene graph nodes. Specifically, it calculates the centroid of the objects mentioned in the text within the visible area of ​​the current map. and the theoretical centroid of the object within the query location range. If the distance between the two is less than a preset threshold If the match is found, the corresponding node is associated; otherwise, the object is determined to be unmatchable in the current map. After completing the node assignment inference, the model uses an autoregressive decoding method to fuse multimodal features and directly predict the two-dimensional coordinates of the target location in the bird's-eye view pixel coordinate system. It then further converts these pixel coordinates into the geographical location in the world coordinate system, achieving accurate text-to-point cloud localization.

[0042] Figure 3 Several examples comparing text-to-point cloud localization using the method of this invention with other methods are presented. The leftmost image shows the input text description, the middle images show the results of various other methods, and the last column on the right shows the results of the method of this invention. Each result image includes the predicted location and the ground truth location, and the prediction error is marked below each image. It can be clearly seen that the method of this invention achieves better localization results under different text descriptions and environments.

[0043] Figure 4 This paper presents a text-to-point cloud cross-modal localization system based on a visual language model, comprising: The point cloud conversion unit 401 is configured to convert the three-dimensional point cloud data to be located into a bird's-eye view image and a scene graph structure, wherein the scene graph structure contains multiple nodes, each node is associated with semantic information and spatial location information in the bird's-eye view image. The multimodal input unit 402 is configured to take the user-input natural language description, bird's-eye view image and scene graph structure as multimodal inputs and provide them to a pre-trained large-scale vision-language model so as to obtain cross-modal features related to the natural language description through the cross-modal interaction mechanism of the large-scale vision-language model. The node association unit 403 is configured to: based on cross-modal features, adopt a partial node association mechanism to selectively map semantic information in the natural language description to at least a portion of the nodes in the scene graph structure, so as to determine the target node consistent with the natural language description; The location determination unit 404 is configured to determine the target location in the 3D point cloud data corresponding to the natural language description based on the spatial location information associated with the target node.

[0044] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0045] According to another embodiment of the present invention, the system of this embodiment can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding method of the present invention on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0046] Figure 5A computer device is shown, which includes a processor 501, a communication interface 502, and a computer-readable storage medium 503. The processor 501, communication interface 502, and computer-readable storage medium 503 can be connected via a bus or other means.

[0047] The communication interface 502 is used to receive and send data. The computer-readable storage medium 503 can be stored in the memory of the electronic device. The computer-readable storage medium 503 is used to store computer programs, which include program instructions. The processor 501 is used to execute the program instructions stored in the computer-readable storage medium 503.

[0048] The processor 501 is the computing and control core of the electronic device. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve the corresponding method flow or corresponding function.

[0049] Processor 501 is configured to perform the following procedure: The 3D point cloud data to be located is converted into bird's-eye view imagery and scene graph structure. The scene graph structure contains multiple nodes, each of which is associated with semantic information and spatial location information in the bird's-eye view imagery. The user-input natural language description, bird's-eye view image, and scene graph structure are used as multimodal inputs and provided to a pre-trained large-scale vision-language model to obtain cross-modal features related to the natural language description through the cross-modal interaction mechanism of the large-scale vision-language model. Based on cross-modal features, a partial node association mechanism is adopted to selectively map the semantic information in the natural language description to at least some nodes in the scene graph structure in order to determine the target node consistent with the natural language description. Based on the spatial location information associated with the target node, determine the target location in the 3D point cloud data corresponding to the natural language description.

[0050] This invention also provides a computer-readable storage medium, which is a memory device in an electronic device for storing programs and data. It is understood that the computer-readable storage medium here may include both built-in storage media in the electronic device and extended storage media supported by the electronic device. The computer-readable storage medium provides storage space for storing the processing system of the electronic device.

[0051] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory; alternatively, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0052] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to perform the following process: The 3D point cloud data to be located is converted into bird's-eye view imagery and scene graph structure. The scene graph structure contains multiple nodes, each of which is associated with semantic information and spatial location information in the bird's-eye view imagery. The user-input natural language description, bird's-eye view image, and scene graph structure are used as multimodal inputs and provided to a pre-trained large-scale vision-language model to obtain cross-modal features related to the natural language description through the cross-modal interaction mechanism of the large-scale vision-language model. Based on cross-modal features, a partial node association mechanism is adopted to selectively map the semantic information in the natural language description to at least some nodes in the scene graph structure in order to determine the target node consistent with the natural language description. Based on the spatial location information associated with the target node, determine the target location in the 3D point cloud data corresponding to the natural language description.

[0053] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0054] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital cable) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text-to-point cloud cross-modal localization method based on a visual language model, characterized in that, Includes the following processes: The 3D point cloud data to be located is converted into a bird's-eye view image and a scene graph structure, wherein the scene graph structure contains multiple nodes, each of which is associated with semantic information and spatial location information in the bird's-eye view image. The user-input natural language description, the bird's-eye view image, and the scene graph structure are used as multimodal inputs and provided to a pre-trained large-scale visual-language model to obtain cross-modal features related to the natural language description through the cross-modal interaction mechanism of the large-scale visual-language model. Based on the cross-modal features, a partial node association mechanism is adopted to selectively map the semantic information in the natural language description to at least some nodes in the scene graph structure, so as to determine the target node consistent with the natural language description; Based on the spatial location information associated with the target node, the target location corresponding to the natural language description in the three-dimensional point cloud data is determined.

2. The text-to-point cloud cross-modal localization method based on a visual language model as described in claim 1, characterized in that, The 3D point cloud data to be located is converted into bird's-eye view imagery and scene graph structure, including: The points in the three-dimensional point cloud data are projected onto the ground plane and rasterized to generate the bird's-eye view image. Based on the object instances in the 3D point cloud data, a node is created for each object instance. The semantic information of the node is the semantic label of the object instance, and the spatial location information of the node is the pixel centroid coordinates of the object instance in the bird's-eye view image, thereby forming the scene graph structure.

3. The text-to-point cloud cross-modal localization method based on a visual language model as described in claim 1 or 2, characterized in that, The user-input natural language description, the bird's-eye view image, and the scene graph structure are used as multimodal inputs and provided to a pre-trained large-scale vision-language model, including: The bird's-eye view image is processed by a visual encoder to obtain visual features; Text features are obtained by processing the natural language description and the scene graph structure through a text encoder; The visual features and the text features are input into the text decoder of the large visual-language model to obtain the cross-modal features.

4. The text-to-point cloud cross-modal localization method based on a visual language model as described in claim 3, characterized in that, Text features are obtained by processing the natural language description and the scene graph structure through a text encoder, including: The semantic information and spatial location information of each node in the scene graph structure are serialized into a text sequence; The text sequence is concatenated with the natural language description to form a joint text input; The text features are obtained by encoding the joint text input using the text encoder.

5. The text-to-point cloud cross-modal localization method based on a visual language model as described in claim 1, characterized in that, Based on the cross-modal features, a partial node association mechanism is employed to selectively map the semantic information in the natural language description to at least a portion of the nodes in the scene graph structure, including: Based on the cross-modal features, the objects mentioned in the natural language description are identified; For each mentioned object, calculate the distance between its theoretical spatial location in the natural language description and the spatial location information associated with each node in the scene graph structure; If the distance is less than a preset threshold, the mentioned object is associated with the corresponding node and used as the target node.

6. The text-to-point cloud cross-modal localization method based on a visual language model as described in claim 1, characterized in that, Based on the spatial location information associated with the target node, determining the target location corresponding to the natural language description in the 3D point cloud data includes: The spatial location information associated with the target node is used as the two-dimensional pixel coordinates in the bird's-eye view image; The target position is obtained by back-projecting the two-dimensional pixel coordinates to the world coordinate system.

7. The text-to-point cloud cross-modal localization method based on a visual language model as described in claim 1, characterized in that, Before using the user-input natural language description, the bird's-eye view image, and the scene graph structure as multimodal input, the method further includes: Obtain a training dataset, which contains multiple samples. Each sample includes a segment of reference 3D point cloud data, a reference bird's-eye view image and a reference scene graph structure generated from the reference 3D point cloud data, and a reference natural language description generated for a specific location in the reference 3D point cloud data. Based on the training dataset, the large-scale vision-language model is fine-tuned and trained. The reference natural language description is automatically generated based on the relative spatial relationship between the specific location and multiple object instances in the reference 3D point cloud data, the semantic category of the object instances, and the color attributes.

8. A text-to-point cloud cross-modal localization system based on a visual language model, characterized in that, include: The point cloud conversion unit is configured to convert the three-dimensional point cloud data to be located into a bird's-eye view image and a scene graph structure, wherein the scene graph structure contains multiple nodes, each of which is associated with semantic information and spatial location information in the bird's-eye view image. The multimodal input unit is configured to take the user-input natural language description, the bird's-eye view image, and the scene graph structure as multimodal inputs and provide them to a pre-trained large-scale visual-language model to obtain cross-modal features related to the natural language description through the cross-modal interaction mechanism of the large-scale visual-language model. The node association unit is configured to: based on the cross-modal features, employ a partial node association mechanism to selectively map the semantic information in the natural language description to at least a portion of the nodes in the scene graph structure, so as to determine the target node consistent with the natural language description; The location determination unit is configured to: determine the target location in the three-dimensional point cloud data corresponding to the natural language description based on the spatial location information associated with the target node.

9. A computer device, characterized in that, include: Processor and computer-readable storage media; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the text-to-point cloud cross-modal localization method based on a visual language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1 to 7: a text-to-point-cloud cross-modal localization method based on a visual language model.

Images