BEV-based scene-level text search point cloud retrieval method and apparatus, and electronic device

Abstract

The invention discloses a BEV-based scene-level text search point cloud retrieval method and device and electronic equipment, and the method comprises the steps: projecting a three-dimensional point cloud into an aerial view, and generating a BEV image; encoding the image by using a pre-trained vision-language model to obtain BEV feature vectors, storing the BEV feature vectors in a vector database, and constructing a point cloud feature library; meanwhile, encoding a user query text to obtain a text feature vector; searching a matching scene from the feature library by calculating the cross-modal similarity between the text and the BEV feature vector; and finally, mapping a retrieval result to original point cloud data and outputting the original point cloud data. By means of the method, the feature alignment problem of the point cloud and the natural language is solved, and the retrieval method which directly and efficiently utilizes essential features of the point cloud data to be in seamless joint with the natural language is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud data retrieval, in particular to a BEV-based scene-level text-to-point cloud retrieval method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of automatic driving, robot navigation and high-precision map construction, three-dimensional laser point cloud as an important data form for perceiving the physical world, its data scale has grown rapidly. How to quickly and accurately retrieve the scene of interest to the user from the massive point cloud scene database has become a key technical challenge. Traditional point cloud retrieval methods rely on artificial pre-set semantic labels or complex point cloud feature matching, which is not only inefficient, but also difficult to understand and respond to flexible natural language query intentions.

[0003] In recent years, with the help of deep learning technology, especially the rise of visual-linguistic pre-training models, it provides a new idea for cross-modal retrieval of "text-to-image". However, directly applying such models to point cloud data faces significant obstacles: the original point cloud is unstructured three-dimensional data, which is very different from images and text in data form. Existing research usually renders point cloud into multi-view two-dimensional images or converts it into voxel grid, but this will cause loss of original three-dimensional spatial relationship and increase of computational complexity, and fails to fully exploit the unique global layout and topology information of the bird's eye view, which is the key to scene-level understanding and retrieval.

[0004] Therefore, there is an urgent need for a retrieval method that can directly and efficiently utilize the essential characteristics of point cloud data and seamlessly integrate with natural language to overcome the limitations of existing technology in alignment efficiency, scene understanding depth and retrieval accuracy. SUMMARY

[0005] The purpose of the present application is to provide a BEV-based scene-level text-to-point cloud retrieval method and device and electronic equipment to solve the problem that visual-linguistic pre-training models cannot be applied to point cloud data.

[0006] According to the first aspect of the embodiments of the present application, a BEV-based scene-level text-to-point cloud retrieval method is provided, comprising: projecting the global three-dimensional point cloud of the target scene to the bird's eye view to generate the corresponding BEV image; encoding the BEV image using a pre-trained visual-linguistic base model to obtain a high-dimensional BEV feature vector; storing the BEV feature vector to a vector database to construct a retrievable point cloud representation library; encoding the search text input by the user using the pre-trained visual-linguistic base model to obtain a text feature vector; calculate the similarity between the text feature vector and each BEV feature vector stored in the vector database, and retrieve the BEV feature vector with a similarity higher than a threshold from the point cloud representation library; According to the BEV feature vector with a similarity higher than a threshold, the corresponding global three-dimensional point cloud data is mapped back and output as a search result.

[0007] According to a second aspect of the embodiments of the present application, a BEV-based scene-level text-to-point cloud retrieval device is provided, comprising: A BEV image generation module is configured to project global three-dimensional point cloud of a target scene to a bird's eye view to generate a corresponding BEV image. A BEV feature encoding module is configured to encode the BEV image using a pre-trained visual-linguistic base model to obtain a high-dimensional BEV feature vector. A vector database construction module is configured to store the BEV feature vector to a vector database to construct a searchable point cloud representation library. A text feature encoding module is configured to encode a search text input by a user using the pre-trained visual-linguistic base model to obtain a text feature vector. A cross-modal retrieval module is configured to calculate the similarity between the text feature vector and each BEV feature vector stored in the vector database, and retrieve the BEV feature vector with a similarity higher than a threshold from the point cloud representation library. A retrieval result output module is configured to map back the corresponding global three-dimensional point cloud data according to the BEV feature vector with a similarity higher than a threshold, and output as a search result.

[0008] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of the first aspect.

[0009] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions, and the instructions are executed by a processor to implement the steps of the method of the first aspect.

[0010] Compared with the prior art, the embodiments of the present application have at least the following beneficial effects: The application projects a three-dimensional point cloud to a bird's eye view, encodes it using a pre-trained visual-linguistic base model, guides the model to focus on the spatial structure and topological relationship in the BEV view using a spatial perception prompt module, and then realizes efficient cross-modal retrieval from text to point cloud by constructing a hierarchical vector database and a multi-granularity similarity calculation strategy. This scheme overcomes the problems of traditional methods relying on manual annotation, not being able to fully utilize the global layout information of the point cloud, and difficulty in cross-modal semantic alignment, significantly improves the accuracy and efficiency of scene-level point cloud retrieval, and reduces the dependence on a large amount of labeled data, providing an effective means for intelligent management and retrieval of massive point cloud data. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of a BEV-based scene-level text-to-point cloud retrieval method according to an example embodiment.

[0012] Figure 2 is a BEV image effect diagram according to an example embodiment.

[0013] Figure 3 is a search result diagram according to an example embodiment.

[0014] Figure 4 is a block diagram of a BEV-based scene-level text-to-point cloud retrieval device according to an example embodiment.

[0015] Figure 5 is a structural schematic diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0016] The specific embodiments of the application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0017] Figure 1 is a flowchart of a BEV-based scene-level text-to-point cloud retrieval method according to an example embodiment. Referring to Figure 1 , the BEV-based scene-level text-to-point cloud retrieval method provided by the embodiment of the application can include: S1: Projecting the global three-dimensional point cloud of the target scene to a bird's eye view to generate a corresponding BEV image; specifically, in the field of autonomous driving, the target scene is usually a global three-dimensional point cloud collected by a vehicle-mounted laser radar, containing roads, vehicles, buildings and traffic facilities. This step aims to convert such road environment point cloud into a bird's eye view suitable for visual model processing. This step includes the following sub-steps: S11: Set the width and height and resolution of the BEV image according to the spatial range of the x-y plane of the global three-dimensional point cloud at a preset scale, and define a regular two-dimensional grid on the horizontal plane; this step is implemented through the following sub-steps: S111: Calculate the spatial boundary of the global three-dimensional point cloud on the horizontal plane (x-y plane).

[0018] Specifically, the x coordinates and y coordinates of all points in the point cloud are traversed to determine the minimum values of the x coordinates and y coordinates respectively 、 and 、 , thereby obtaining an axially aligned two-dimensional boundary rectangular frame.

[0019] S112: Calculate the size of the BEV image based on a preset projection scale and a target resolution. The preset scale is 1:100, i.e., 1 centimeter in the real world corresponds to one pixel in the BEV image, then the width of the BEV image is , where ceil is the ceiling function.

[0020] S113: On the horizontal plane, define a regular two-dimensional grid covering the two-dimensional boundary rectangular frame. The grid is composed of square cells, and the physical side length of each grid cell is 1 centimeter, which corresponds to one pixel of the BEV image. The origin of the grid is aligned with the minimum corner point of the boundary frame.

[0021] S12: Project the global three-dimensional point cloud along the vertical direction to the two-dimensional grid, and encode the intensity information of the point cloud in each grid cell into RGB information to obtain a BEV image; this step is implemented through the following sub-steps: S121: Perform vertical projection. Traverse each point in the global three-dimensional point cloud, where is the intensity value of the point. According to its coordinates , calculate the grid cell to which it belongs : where floor is the floor function, and mark the point P as falling into the grid cell with index .

[0022] S122: Process the point cloud intensity information in each grid cell. For each non-empty grid cell, collect the intensity value set of all points falling into it. Calculate the representative intensity value i.e. the set the most frequent intensity value.

[0023] S123: encode the intensity values as RGB information. Set the effective range of intensity values . First linearly normalize the representative intensity values : Then map the normalized intensity values to the corresponding (R, G, B) triplets as the color values of the pixels corresponding to the grid cell. For the empty grid cells that do not fall into any point, their pixel color values are set to (0, 0, 0), i.e. black.

[0024] S124: synthesize the BEV image. According to the pixel color values corresponding to all grid cells, generate an RGB three-channel digital image with a width of and a height of , i.e. the final BEV image, as shown in Figure 2 , in which different objects such as road surface, road surface markings, and sidewalks present different colors or brightness due to their different reflection intensities.

[0025] Since the original three-dimensional point cloud collected by autonomous driving is unstructured massive data, direct processing has high computational complexity and is difficult to be compatible with the visual model designed for images. This step aims to convert the three-dimensional point cloud into a two-dimensional representation that can not only preserve the core spatial information of the road scene (such as the direction of lane lines and road surface markings), but also conform to the input format of the standard visual model. By projecting to the bird's eye view, this method maximally preserves the global layout and topological relationship of the scene in the horizontal plane, which is the key to scene-level understanding. At the same time, using point cloud intensity information to encode RGB can implicitly distinguish objects with different reflection characteristics in the image, providing a rich and structured input for subsequent feature extraction, and fundamentally solving the problem of point cloud data being difficult to be directly used for efficient visual representation learning.

[0026] S2: encode the BEV image using a pre-trained visual-linguistic base model to obtain a high-dimensional BEV feature vector; Specifically, in order to extract deep semantic features that can be aligned with natural language descriptions from the BEV image of the autonomous driving road scene, the present application adopts a pre-trained visual-linguistic base model, which includes a visual encoder, a text encoder, a spatial perception prompt module, and an adaptive pooling fusion module. The BEV image passes through the following modules of the visual-linguistic base model to obtain a BEV feature vector: (1) visual encoder; In particular, the visual encoder is a pre-trained deep convolutional neural network with cross-modal alignment capability. The BEV image is first passed through a backbone network composed of multiple convolutional layers and down-sampling layers to extract multi-scale spatial feature maps. In particular, the backbone network outputs a sequence of feature maps from shallow to deep , where the shallow feature maps contain rich edge and texture information, and the deep feature maps contain higher-level semantic information.

[0027] (2) a spatial perception hinting module; In particular, to enable the model to better understand the spatial structure specific to the BEV perspective of autonomous driving, the module performs the following A1-A3 processing: A1: generating a spatial perception hinting vector. The spatial perception hinting module maintains a set of trainable parameter matrices P. According to the size of the input BEV image and the network structure, the module generates a set of two-dimensional hinting vector maps that match the spatial dimensions (width and height) of a certain intermediate layer feature map (e.g. ) in the visual encoder.

[0028] A2: extracting spatial prior knowledge representation. From the spatial perception hinting module, extract the learned parameters P or the generated hinting vector maps encode abstract knowledge. By performing global average pooling on P or , and passing through a small projection network, a fixed-dimensional, modality-independent vector representation is obtained, which is the vivid knowledge representation of the spatial structure and topological relationship . This representation aims to capture general road structures, object layouts, and other spatial priors in the BEV perspective.

[0029] A3: injecting the hinting vector and modulating the feature. Perform element-wise addition operation between the generated hinting vector map and the intermediate layer feature map to obtain the modulated feature map . This operation directly incorporates prior knowledge about road structures, object layouts, and topological relationships in the BEV perspective into the representation of visual features as a soft constraint, guiding the network to focus on these spatial information highly relevant to scene understanding in subsequent layers.

[0030] (3) an adaptive pooling fusion module; In particular, to generate a global scene-level representation, the module performs the following B1-B3 processing: B1: Extract multi-level modulation features. From multiple different depth layers of the visual encoder (including the layer of the modulated features and its subsequent layers), a set of representative feature maps is selected, denoted as set .

[0031] B2: Perform adaptive feature fusion. The adaptive pooling integration module performs an adaptive spatial pooling operation on each feature map in set , and compresses the feature maps of different sizes into fixed-length feature vectors .

[0032] B3: Aggregate global representation. All fixed-length feature vectors are spliced to form a new composite feature vector. Then, a fully connected layer is used to reduce and fuse the composite vector, and finally a global high-dimensional vector with scene-level representation capability, i.e., the BEV feature vector is output.

[0033] (4) Text encoder; Specifically, the text encoder is a pre-trained Transformer network with cross-modal alignment capability. When the user inputs a search text or an expanded structured description sequence, the text encoder first performs tokenization and embedding on the input text, and then encodes it through multiple layers of self-attention mechanism, finally outputting a high-dimensional vector that can represent the overall text semantics . The text encoder has been aligned with the visual encoder through contrastive learning in the pre-training phase, so the output of the text encoder and the output of the visual encoder are in a shared semantic embedding space, so that the features of the two modalities can be directly compared in similarity.

[0034] The purpose of the above text encoder is to extract deep features from the road scene BEV image that contain both visual details and high-level semantics, and can be aligned with the text modality. It is difficult to focus on the spatial structure specific to the BEV perspective using a general-purpose visual model, but by introducing a spatial perception prompt module, the model is explicitly guided to pay attention to spatial elements such as lanes, intersections, and vehicle layouts that are crucial to scene retrieval. Then, through the adaptive pooling fusion module, multi-scale features are aggregated, and the final feature vector has strong scene-level representation capability. At the same time, the model achieves cross-modal alignment through pre-training, so that its visual encoding and text encoding share a semantic space, laying a solid foundation for subsequent image-text similarity calculation.

[0035] S3: store the BEV feature vector into a vector database to build a searchable point cloud representation library; in order to manage the massive point cloud feature data of different road scenes (such as highway, urban area, tunnel) accumulated in the field of autonomous driving and support efficient retrieval, a special point cloud representation library needs to be built. This step includes the following sub-steps: S31: build a hierarchical prunable index structure in the vector database to form the retrieval architecture of the point cloud representation library.

[0036] Index top layer construction: for each BEV feature vector to be stored, one or more predefined semantic attribute labels are associated, such as "drivable road", "sidewalk", "parking lot", etc., which are derived from the annotation of the BEV image. The top layer of the index divides all the BEV feature vectors of the scenes according to these semantic attribute labels, organizes the feature vectors with the same or similar main semantic attributes under the same top layer clustering node, and realizes fast and coarse-grained filtering of the whole library scenes.

[0037] Index bottom layer construction: within each top layer semantic clustering node, a vector similarity-based clustering algorithm is used to perform secondary fine-grained clustering and organization of all vectors in the group according to the cosine distance of the BEV feature vectors in the high-dimensional space. The bottom layer index structure is designed to be prunable, allowing dynamic adjustment of the breadth and depth of the search according to the trade-off between accuracy and speed during retrieval.

[0038] S32: use a hybrid partitioning strategy and on-demand loading mechanism in the vector database to optimize the point cloud representation library.

[0039] Hot data memory cache: the system continuously monitors the access frequency of different scenes in the point cloud representation library. The BEV feature vectors corresponding to the scenes (such as "urban intersection", "open parking lot") with access frequency exceeding a preset threshold within a certain period are determined as "hot spot vectors". The system loads and stores the data copies of these hot vectors and the key nodes in their hierarchical index into the high-speed index structure in the memory, ensuring rapid response to high-frequency queries.

[0040] Full data persistent storage and dynamic loading: all BEV feature vectors and their complete hierarchical index structure in the point cloud representation library are persistently saved to the persistent vector storage system closely associated with the vector database service. When non-hot spot scene retrieval or index traversal is needed, the system uses the on-demand loading mechanism to dynamically load the relevant index nodes and corresponding feature vectors from the persistent storage to the memory for calculation, thereby realizing efficient management of the super-large-scale point cloud representation library under limited memory resources.

[0041] This step constructs a hierarchical, pruneable index, achieving a qualitative leap in retrieval efficiency, enabling rapid coarse screening followed by fine-grained comparison. A hybrid partitioning strategy intelligently balances access speed and storage capacity within limited memory resources. This mechanism collectively builds a point cloud representation library supporting large-scale, high-performance retrieval, giving the system scalability for practical applications.

[0042] S4: Encode the user-input search text using the pre-trained vision-language base model to obtain a text feature vector; specifically, when a user wants to retrieve a specific road scene from the autonomous driving point cloud library using natural language, such queries need to be encoded into point cloud feature-aligned vectors. This step includes the following sub-steps: S41: Through the text restatement and contextual enhancement module, the natural language query input by the user is parsed and expanded into a structured description sequence that can clearly describe the spatial structure, object layout and global topological relationship contained in the BEV image, so as to obtain an enhanced text representation.

[0043] Specifically, the internal processing flow of the text restatement and contextual enhancement module is as follows: S411: Perform semantic parsing and standardization on the input natural language query. For example: "There is a T-junction with a parking area on the right side of the main road". First, use dependency parsing to identify the core entities ("T-junction", "main road", "parking area") and the spatial relation predicate ("right side").

[0044] S412: Based on a predefined BEV scene description ontology library, the parsed results are expanded and structurally reorganized. This ontology library defines road elements, traffic facilities, static objects, dynamic objects, and their typical spatial relationships from a BEV perspective, mapping and expanding the identified entities and relationships into standard structured description sequences. For example, the above query is expanded into the sequence: "[Scene Type: Road Intersection]; [Intersection Shape: T-shaped]; [Element 1: Main Road]; [Element 2: Parking Area]; [Spatial Relationship: Parking area is located on the right side of the main road]". This sequence explicitly enhances the spatial structure and topological relationships from a BEV perspective.

[0045] S413: Convert the generated structured description sequence into a standardized text format suitable for subsequent text encoder input, i.e., the enhanced text representation. .

[0046] S42: The enhanced text representation is input into the text encoder of the vision-language base model, and prior knowledge of spatial structure and topological relationships provided by the spatial awareness cueing module is simultaneously injected to obtain text encoding that is initially aligned with the BEV visual feature space; this step is implemented through the following sub-steps: S421: Text encoder encoding. The enhanced text representation is then encoded. The input is fed into the text encoder. The text encoder processes the input sequence through its internal embedding layers and multi-layer self-attention mechanisms to generate an initial context-aware text feature representation. .

[0047] S422: Cross-modal prior knowledge injection. To achieve alignment between text-side and visual-side spatial understanding, prior knowledge representations regarding the spatial structure and topological relationships of the BEV scene are obtained from the learnable cue vectors generated by the spatial perception cue module. Through a feature fusion layer, This modulated text feature representation is injected into the intermediate layer output of the text encoder or the final text feature representation to generate the modulated text feature representation. This step aims to subject the text encoding process to the same spatial structure prior as visual encoding. S423: Outputs the initially aligned text encoding. Further processing (such as using an output vector corresponding to a specific [CLS] tag or pooling) generates a fixed-dimensional vector representation, namely the text encoding initially aligned with the BEV visual feature space. .

[0048] S43: The text encoding of the initial alignment is fine-tuned using a text-scene alignment loss based on contrastive learning to generate a final text feature vector that shares a highly consistent cross-modal semantic space with the BEV feature vector. During the model training phase, a text encoding based on the initial alignment is constructed. and its corresponding matched BEV feature vector Positive sample pairs are formed, and multiple pairs are constructed simultaneously. Negative sample pairs are formed by combining positive sample pairs with non-matching BEV feature vectors. The contrastive learning loss function infoNCE is employed, which aims to maximize the cosine similarity of positive sample pairs in the vector space while minimizing the similarity with all negative sample pairs. The parameters of the text encoder and related fusion layers are optimized using the backpropagation algorithm. In semantic space The vectors converge and distance themselves from vectors unrelated to the scene. After this optimization process, the final output vector of the text encoder for the input query is the final text feature vector that shares a highly consistent cross-modal semantic space with the BEV feature vector. This vector can be directly used for efficient similarity retrieval with BEV feature vectors in the point cloud representation library.

[0049] The user-input search text is encoded by the text rephrasing and scene enhancement module, which structures and clarifies the natural language query, making it describe the content of the BEV image. Crucially, the text encoding process shares the understanding of the BEV spatial structure with the visual side by injecting homologous spatial prior knowledge . After fine-tuning based on contrastive learning, the final generated text feature vector is deeply constrained in the cross-modal semantic space consistent with , greatly improving the semantic comparability between the text query and the point cloud scene.

[0050] S5: Calculate the similarity between the text feature vector and each BEV feature vector stored in the vector database, and retrieve the BEV feature vector with a similarity higher than the threshold from the point cloud representation library. Specifically, after receiving the encoded text feature vector, the system needs to quickly and accurately find the matching road scene in the vast automatic driving point cloud representation library. This step performs efficient and accurate two-stage search based on the hierarchical index and storage architecture that has been built, and the specific implementation includes the following sub-steps: S51: Perform a two-stage progressive retrieval strategy, first perform coarse screening based on semantic attributes at the top layer of the hierarchical index of the vector database to recall a candidate scene set; According to the query semantics corresponding to the text feature vector , the system first operates at the top layer of the hierarchical index of the vector database. This top layer index is organized according to the semantic attributes of the scene (such as "urban road" and "parking lot"). The system calculates the matching degree of the query semantics and the center vector of each top-level semantic clustering node, quickly filters out a number of top-level clustering nodes that are most relevant, and preliminarily aggregates all scene identifiers under these nodes to form a large-scale candidate scene set . The goal of this stage is to quickly exclude a large number of obviously irrelevant scenes and achieve coarse-grained screening to narrow the search range.

[0051] S52: Calculate the multi-granularity cross-modal similarity between the text feature vector and each BEV feature vector stored in the vector database for the candidate scene set at the bottom layer of the hierarchical index of the vector database; At the bottom layer of the hierarchical index of the vector database, perform multi-granularity cross-modal similarity calculation for the BEV feature vector corresponding to each scene in the candidate scene set and the text feature vector . The calculation process is as follows: S521: Calculate the cosine similarity between and Each feature vector is input to a shared projection layer (a small multilayer perceptron, MLP). This projection layer maps the feature vectors from the two different modalities into a common, lower-dimensional semantic subspace, resulting in the projected vectors. and Then, the cosine similarity between the two projection vectors is calculated. : Cosine similarity It serves as a global similarity metric for measuring the semantic matching degree of the overall scene.

[0052] S522: Fine-grained semantic alignment is achieved through a lightweight interactive attention mechanism. First, the aggregated data is generated... Intermediate layer modulation feature map As a set of local region features of BEV. As the query vector, its cross-attention weight with the feature vectors of each region in the local region feature set is calculated. Subsequently, the feature set of the local region is weighted and aggregated according to this weight to generate a visual context vector aligned with the semantics of the text. Finally, calculate. and cosine similarity : Cosine similarity The local semantic alignment score is used to measure the degree of alignment between the text description and the local details of the scene.

[0053] S523: The global similarity measure and the local semantic alignment score obtained above are weighted and fused to obtain the final multi-granularity cross-modal similarity. : Where λ is a preset fusion weight parameter (0≤λ≤1). This similarity combines scene-level overall semantic matching with fine-grained local semantic alignment information.

[0054] S53: Reorder the candidate scene set according to the multi-granularity cross-modal similarity, and output the BEV feature vectors with the final retrieved similarity higher than the threshold according to the dynamic threshold strategy. After calculating the candidate scene set The final multi-granularity cross-modal similarity for each scene Afterwards, according to The system reorders all candidate scenes in descending order based on their numerical values. A dynamic threshold strategy is used to determine the returned result. Specifically, the dynamic threshold can be set to the top K similarity values ​​in the sorted list (K is a preset value). Based on this strategy, the system outputs all similarity values ​​from the sorted list. BEV feature vectors that are higher than (or equal to) this dynamic threshold This will be the final result of this text search.

[0055] The proposed similarity calculation scheme employs a two-stage progressive retrieval strategy, fully utilizing the index structure. First, it quickly eliminates a large number of irrelevant scenarios based on semantic attributes, significantly narrowing the search scope. In the core similarity calculation, it innovatively uses multi-granularity cross-modal similarity, integrating a global similarity metric reflecting overall semantic matching with a local semantic alignment score capturing detailed alignment, making similarity assessment more comprehensive and accurate. A dynamic threshold strategy ensures the flexibility and reliability of the returned result set, achieving an optimal balance between retrieval accuracy and efficiency.

[0056] S6: Map the BEV feature vectors with similarity higher than the threshold back to the corresponding global 3D point cloud data and output them as search results; specifically, the retrieved BEV feature vectors themselves are not the final target, and users need to obtain complete and analyzable 3D road scene point cloud data. This step includes the following sub-steps: S61: Based on the BEV feature vectors with similarity higher than the threshold, parse the corresponding scene identifier from the vector database; Specifically, after completing the retrieval and obtaining a set of BEV feature vectors with similarity higher than a threshold... Then, based on the indexes and mapping relationships established in the vector database, the following steps are performed: Each vector in the database undergoes reverse parsing. Specifically, when storing each BEV feature vector, the vector database binds it to a scene identifier that uniquely identifies its source scene. The system queries the database's metadata index to perform reverse parsing for each resulting vector. Parse the corresponding scene identifier The system uses the parsed scene identifier Access a persistent storage system for raw point cloud data that is independent of the vector database.

[0057] S62: Based on the scene identifier, load the complete global 3D point cloud data from the independent persistent storage system that stores the original point cloud data; In this storage system, the raw 3D point cloud data is organized by scene and addressed using scene identifiers. The system loads data by calling the corresponding data read interface. corresponding, complete global three-dimensional point cloud data file. Finally, the system will load one or more three-dimensional point cloud scenes as the final search result of this "text searching point cloud" retrieval, as shown in Figure 3 indicated, and provide the result to the user for visualization, analysis or further processing.

[0058] The scene identifier efficiently connects the feature vector library for fast retrieval and the original point cloud library storing complete information. This "feature-original data" separation architecture design ensures the speed of the retrieval stage and ensures the completeness and high availability of the final output result, providing the user with intuitive and operable three-dimensional scene retrieval results, realizing seamless conversion from "text searching feature" to "text searching point cloud".

[0059] From the above technical solutions, the application realizes efficient cross-modal retrieval from text to point cloud by projecting three-dimensional point cloud to bird's eye view and using a pre-trained visual-language base model for encoding, combining a spatial perception prompt module to guide the model to focus on the spatial structure and topological relationship in the BEV perspective, and then constructing a hierarchical vector database and a multi-granularity similarity calculation strategy. This scheme overcomes the problems of traditional methods relying on manual annotation, not being able to fully utilize the global layout information of point cloud, and difficulties in cross-modal semantic alignment, significantly improves the accuracy and efficiency of scene-level point cloud retrieval, and reduces the dependence on a large amount of labeled data, providing an effective means for intelligent management and retrieval of massive point cloud data.

[0060] Figure 4 is a block diagram of a BEV-based scene-level text searching point cloud retrieval device according to an exemplary embodiment. Referring to Figure 4 , the device includes: a BEV image generation module 1 for projecting the global three-dimensional point cloud of a target scene to a bird's eye view to generate a corresponding BEV image; a BEV feature encoding module 2 for encoding the BEV image using a pre-trained visual-language base model to obtain a high-dimensional BEV feature vector; a vector database construction module 3 for storing the BEV feature vector into a vector database to construct a searchable point cloud representation library; a text feature encoding module 4 for encoding the search text input by the user using the same visual-language base model to obtain a text feature vector; a cross-modal retrieval module 5 for calculating the similarity between the text feature vector and each BEV feature vector stored in the vector database, and retrieving the BEV feature vectors with a similarity higher than a threshold from the point cloud representation library; The search result output module 6 is configured to map the BEV feature vectors with the similarity higher than the threshold back to the corresponding global three-dimensional point cloud data and output as the search result.

[0061] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the BEV-based scene-level text search point cloud search method as described above. As Figure 5 As shown in the figure, a hardware structure diagram of an apparatus for BEV-based scene-level text search point cloud search provided by an embodiment of the present application is located in any device with data processing capability. In addition to the Figure 5 In addition to the processor and the memory, any device with data processing capability where the apparatus is located in the embodiment can also include other hardware according to the actual functions of the device with data processing capability, and no further description is given.

[0062] Correspondingly, the present application also provides a computer readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the BEV-based scene-level text search point cloud search method as described above. The computer readable storage medium can be an internal storage unit of any device with data processing capability, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of the wind power generator, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any device with data processing capability and the external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the device with data processing capability, and can also be used to temporarily store data that has been output or will be output.

[0063] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary only. It is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. It is intended to cover and invoke the equivalents of all features of the application disclosed and claimed herein together with full equivalents of the features presented throughout this disclosure incorporated by reference.

[0064] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that changes and modifications can be effected therein by those skilled in the art without departing from the scope of the application. The scope of the application should be determined only by the appended claims.

Claims

1. A scene-level text search point cloud retrieval method based on BEV (Browser Emission Vehicle), characterized in that, include: The global 3D point cloud of the target scene is projected onto the bird's-eye view to generate the corresponding BEV image; The BEV image is encoded using a pre-trained vision-language base model to obtain a high-dimensional BEV feature vector. The BEV feature vectors are stored in a vector database to construct a searchable point cloud representation library. The pre-trained visual-language base model is used to encode the search text input by the user to obtain a text feature vector; Calculate the similarity between the text feature vector and each BEV feature vector stored in the vector database, and retrieve BEV feature vectors with similarity higher than a threshold from the point cloud representation library; The BEV feature vectors with similarity higher than the threshold are mapped back to the corresponding global 3D point cloud data and output as the search results.

2. The method according to claim 1, characterized in that, Projecting the global 3D point cloud of the target scene onto the bird's-eye view generates the corresponding BEV image, including: Based on the spatial range of the xy plane of the global 3D point cloud, the width, height and resolution of the BEV image are set according to a preset ratio, and a regular two-dimensional grid is defined on the horizontal plane. The global 3D point cloud is projected vertically onto the 2D grid, and the intensity information of the point cloud in each grid cell is encoded into RGB information to obtain a BEV image.

3. The method according to claim 1, characterized in that, The vision-language basic model includes: a visual encoder, a text encoder, a spatial awareness cueing module, and an adaptive pooling fusion module, wherein: Both the visual encoder and the text encoder are pre-trained neural networks with cross-modal alignment capabilities. The visual encoder is used to extract hierarchical features from the input BEV image; The text encoder is used to encode the input text sequence and extract text semantic features. Its output representation is constrained in a semantic embedding space shared with visual features. The spatial perception cue module is used to generate learnable spatial perception cue vectors and inject the spatial perception cue vectors into the intermediate layer of the visual encoder to modulate the visual features and make them focus on the spatial structure and topological relationship from the BEV perspective. The adaptive pooling fusion module is used to extract the modulated visual features and aggregate them into a global, high-dimensional BEV feature vector with scene-level representation capabilities.

4. The method according to claim 1, characterized in that, The BEV feature vectors are stored in a vector database to construct a searchable point cloud representation library, including: In the vector database, a hierarchical pruning index structure is constructed, in which the top layer performs coarse clustering based on the semantic attributes of the scene, and the bottom layer performs fine-grained organization based on the similarity of the BEV feature vectors, so as to form the retrieval architecture of the point cloud representation library. In the vector database, a hybrid partitioning strategy is adopted, storing frequently accessed hot scene vectors in an in-memory index, while all BEV feature vectors are stored in a persistent vector storage associated with the vector database and managed through an on-demand loading mechanism to optimize the point cloud representation library.

5. The method according to claim 1, characterized in that, The search text input by the user is encoded using the aforementioned visual-language basic model to obtain a text feature vector, including: Through the text restatement and contextual enhancement module, the natural language query input by the user is parsed and expanded into a structured description sequence that can clearly describe the spatial structure, object layout and global topological relationship contained in the BEV image, so as to obtain an enhanced text representation. The enhanced text representation is input into the text encoder of the visual-language base model, and prior knowledge of spatial structure and topological relationships provided by the spatial awareness prompting module is injected at the same time to obtain text encoding that is initially aligned with the BEV visual feature space. The initially aligned text encoding is fine-tuned using a text-scene alignment loss based on contrastive learning to generate a final text feature vector that shares a highly consistent cross-modal semantic space with the BEV feature vector.

6. The method according to claim 1, characterized in that, Calculating the similarity between the text feature vector and each BEV feature vector stored in the vector database, and retrieving BEV feature vectors with similarity higher than a threshold from the point cloud representation library, including: A two-stage progressive retrieval strategy is implemented. First, a coarse screening based on semantic attributes is performed at the top level of the hierarchical index of the vector database to recall the candidate scenario set. At the bottom layer of the hierarchical index of the vector database, the multi-granularity cross-modal similarity between the text feature vector and each BEV feature vector stored in the vector database is calculated for the candidate scene set. The candidate scene set is reordered based on the multi-granularity cross-modal similarity, and the BEV feature vectors with similarity higher than the threshold are output according to the dynamic threshold strategy.

7. The method according to claim 6, characterized in that, Calculating the multi-granularity cross-modal similarity between the text feature vector and each BEV feature vector stored in the vector database includes: The text feature vector and a BEV feature vector are respectively mapped to a common semantic subspace through a shared projection layer, and the cosine similarity between the two in the subspace is calculated as a global similarity measure. A lightweight interactive attention mechanism is used to calculate the cross-attention weights between the text feature vector and the BEV feature vector in different local regions. Based on these weights, a visual context vector aligned with the text semantics is obtained. Then, the matching score between the context vector and the text feature vector is calculated as the local semantic alignment score. The global similarity metric and the local semantic alignment score are weighted and fused to obtain the final multi-granularity cross-modal similarity.

8. A scene-level text search point cloud retrieval device based on BEV, characterized in that, include: The BEV image generation module is used to project the global 3D point cloud of the target scene onto the bird's-eye view to generate the corresponding BEV image. The BEV feature encoding module is used to encode the BEV image using a pre-trained vision-language base model to obtain a high-dimensional BEV feature vector. The vector database construction module is used to store the BEV feature vectors into a vector database to construct a searchable point cloud representation library. The text feature encoding module is used to encode the user-input search text using the pre-trained visual-language basic model to obtain a text feature vector; The cross-modal retrieval module is used to calculate the similarity between the text feature vector and each BEV feature vector stored in the vector database, and to retrieve BEV feature vectors with similarity higher than a threshold from the point cloud representation library. The search result output module is used to map the BEV feature vectors with similarity higher than the threshold back to the corresponding global 3D point cloud data and output them as search results.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images