A fine-grained visual language alignment contrast learning implementation method and device and medium

Abstract

The application discloses a fine-grained visual language alignment contrast learning implementation method and device and a medium, and belongs to the technical field of point cloud open vocabulary segmentation. The method uses a mask to continuously mask regional point cloud data, and constructs a mask feature beneficial to model learning of point cloud data, so that discriminative visual information can be retained; a fine-grained text language alignment training method is adopted, so that the model can learn the correlation between visual language in a finer granularity, the matching degree of the point cloud model feature space and the text feature space is improved, and finally the performance of semantic segmentation of the point cloud segmentation model in a non-labeled scene is realized. The application solves the open vocabulary point cloud semantic segmentation problem in a non-labeled and very complex point cloud structure scene, can be widely used in large-scale 3D visual understanding application scenes, breaks the strong dependence of a full-supervised point cloud segmentation method on dense and expensive labeling, and increases the application range of the point cloud segmentation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and particularly relates to a fine-grained visual language alignment contrast learning implementation method and device and medium. BACKGROUND

[0002] In the field of visual scene understanding, deep neural networks have made impressive progress in various core tasks such as segmentation, tracking and detection. However, most methods are based on the closed set assumption, and the model can only recognize pre-defined classes in the training set and does not have the ability to recognize unseen classes. These limitations may increase the difficulty and cost of implementing deep models on new scenes, especially when the number of classes or concepts in the scene is much larger than the number contained in the training data set, which hinders the application of many real scenes. In recent years, due to the rapid development of visual language pre-training, open vocabulary understanding has been proposed. These new methods attempt to locate and identify classes outside the annotation label space.

[0003] Point cloud semantic segmentation is a basic task in three-dimensional computer vision, which aims to assign a class to each point in the point cloud scene. Traditional three-dimensional semantic segmentation methods usually train the model in a closed set data set and can only perceive the trained classes. In recent years, open vocabulary semantic segmentation has attracted more and more attention, and the model is trained to recognize a set of open object classes, so it can be directly used for various applications. This ability is provided by the common feature space between language and vision. Visual language base models trained on large-scale image language data sets can bridge text and visual information by mapping visual and text features to the same feature space, thereby understanding the mutual relationship between language and visual information.

[0004] A crucial step towards open-vocabulary segmentation of point clouds is aligning 3D visual features with corresponding linguistic features. Existing methods can be broadly categorized into two types: dense alignment and region alignment. Dense alignment seeks fine-grained alignment between points in a point cloud and pixels in text or images. These methods acquire point-text or point-pixel pairs with the aid of a 2D segmentation model, then align each point feature with its corresponding text or pixel feature. However, due to calibration errors, noisy registration between points and pixels leads to the model learning mismatched representations and losing 3D spatial information. Region alignment methods follow the CLIP (Contrastive Language-Image Pre-training) paradigm, aligning aggregated point cloud region features with features of the corresponding description. Region point clouds can be obtained by downsampling scene point clouds, depth image calibration, or the scene point cloud itself. Corresponding text descriptions are obtained using a 2D model. The region point clouds are fed into a 3D backbone network to extract features, and the text descriptions are input into a frozen text encoder to extract linguistic features. Point cloud region features and language features are pooled into a single vector with the same dimension. Contrastive learning is then applied to align point cloud region features with language features, and this approach has yielded promising results. However, pooling compresses both point cloud region features and language features, which may lead to the loss of discriminative information, resulting in coarse visual-language alignment.

[0005] Therefore, new methods are urgently needed to find ways to preserve discriminative features to the greatest extent possible during the alignment process and to achieve more fine-grained visual-language alignment. Summary of the Invention

[0006] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, apparatus and medium for fine-grained visual language alignment and contrast learning for point cloud open vocabulary segmentation.

[0007] The technical solution adopted in this invention is:

[0008] A fine-grained visual language alignment and contrast learning implementation method includes the following steps:

[0009] Acquire raw point cloud data, which includes point cloud data of a preset area and corresponding text descriptions;

[0010] Visual mask data augmentation is performed on the original point cloud data to obtain visual mask data augmented point cloud data;

[0011] A point cloud segmentation model is established. The original point cloud data and the visual mask data-enhanced point cloud data are input into the point cloud segmentation model to obtain the point-by-point features of the original point cloud and the point-by-point features of the visual mask data-enhanced point cloud.

[0012] perform feature extraction on the text description to obtain word-by-word text features and text overall features;

[0013] generate a first contrastive learning loss according to the average point cloud features and the text overall features;

[0014] generate a second contrastive learning loss according to the average point cloud features and the word-by-word text features;

[0015] generate a third contrastive learning loss according to the point-by-point features of the original point cloud and the word-by-word text features;

[0016] generate a consistency loss according to the point-by-point features of the original point cloud and the visually masked data enhanced point cloud features;

[0017] train the point cloud segmentation model according to the first contrastive learning loss, the second contrastive learning loss, the third contrastive learning loss and the consistency loss.

[0018] Further, the visual mask data augmentation on the original point cloud data to obtain visually masked data enhanced point cloud data comprises:

[0019] obtain a series of cuboids for dividing the spatial data of the original point cloud data according to the original point cloud data and a region of a set size;

[0020] obtain a series of cuboid subsets to be masked according to a set mask ratio and the cuboids;

[0021] mask all original point cloud data in the cuboid subsets.

[0022] Further, the first contrastive learning loss L CTA is expressed as:

[0023]

[0024] wherein B represents the number of samples in a batch of data, represents the average point cloud features from the i-th sample, represents the text overall features from the i-th sample, and τ represents a temperature coefficient in contrastive learning.

[0025] Further, the second contrastive learning loss L wEA is expressed as:

[0026]

[0027] wherein B represents the number of samples in a batch of data, represents the average similarity of the text global feature from the i-th sample and the point cloud feature from the j-th sample.

[0028] Further, the third contrastive learning loss L PEA is expressed as:

[0029]

[0030] In the formula, B represents the number of samples in a batch of data, represents the average similarity of the text global feature from the i-th sample and the point cloud feature from the j-th sample.

[0031] Further, the consistency loss L consis is expressed as:

[0032]

[0033] In the formula, JS(·,·) represents a divergence function that minimizes the difference between features, n represents the n-th point cloud data, Z, Z m respectively, are the original point cloud point-by-point features and the visual mask data enhanced point cloud point-by-point features; N represents a set composed of point cloud data, and [·] represents an indexing operation.

[0034] Further, the expression of the loss function for training the point cloud segmentation model is:

[0035] L=αL CTA +βL WEA +γL PEA +δL consis

[0036] In the formula, L CTA is the first contrastive learning loss, L WEA is the second contrastive learning loss, L PEA is the third contrastive learning loss, L consis is the consistency loss; α, β, γ, δ are all hyperparameters.

[0037] The second technical solution adopted by the present application is:

[0038] A fine-grained visual language alignment contrast learning implementation device, comprising:

[0039] A data acquisition module is configured to acquire original point cloud data, wherein the original point cloud data includes point cloud of a preset area and corresponding text description.

[0040] A point cloud enhancement module is configured to perform visual mask data enhancement on the original point cloud data to obtain visual mask data enhanced point cloud data.

[0041] The point cloud segmentation module is configured to establish a point cloud segmentation model, input the original point cloud data and the visual mask data enhanced point cloud data into the point cloud segmentation model, and obtain original point cloud point-by-point features and visual mask data enhanced point cloud point-by-point features.

[0042] The feature extraction module is configured to perform feature extraction on the text description, and obtain word-by-word text features and overall text features.

[0043] The first loss module is configured to obtain average point cloud features according to the original point cloud point-by-point features, and generate a first contrastive learning loss according to the average point cloud features and the overall text features.

[0044] The second loss module is configured to generate a second contrastive learning loss according to the average point cloud features and the word-by-word text features.

[0045] The third loss module is configured to generate a third contrastive learning loss according to the original point cloud point-by-point features and the word-by-word text features.

[0046] The fourth loss module is configured to generate a consistency loss according to the original point cloud point-by-point features and the visual mask data enhanced point cloud point-by-point features.

[0047] The model training module is configured to train the point cloud segmentation model according to the first contrastive learning loss, the second contrastive learning loss, the third contrastive learning loss, and the consistency loss.

[0048] The third technical solution adopted by the present application is:

[0049] The fine-grained visual language alignment contrastive learning implementation device comprises:

[0050] At least one processor;

[0051] At least one memory for storing at least one program;

[0052] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0053] The fourth technical solution adopted by the present application is:

[0054] A computer readable storage medium, wherein a processor executable program is stored, the processor executable program is used for executing the above method when executed by a processor.

[0055] Compared with the prior art, the present application has the beneficial effects that the present application provides a fine-grained visual language alignment method for point cloud open vocabulary segmentation tasks, which retains discriminative visual language information and effectively improves the segmentation performance of the point cloud segmentation model under no annotation.

[0056] Firstly, the region point cloud data is masked for continuity, and a mask feature beneficial to model learning point cloud data context is constructed, so as to retain discriminative visual information; secondly, the fine-grained text language alignment training method proposed in the application is adopted, so that the model can learn the correlation between visual and language in a finer granularity, improve the matching degree of the point cloud model feature space and the text feature space, and finally realize the performance of semantic segmentation of the point cloud segmentation model in the unannotated scene. The application effectively solves the open vocabulary point cloud semantic segmentation problem in the unannotated and complex point cloud structure scene, and can be widely used in large-scale 3D visual understanding application scenes such as automatic driving and city detection, breaking the strong dependence of full-supervised point cloud segmentation method on dense and expensive annotation, and greatly increasing the application range of point cloud segmentation model. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0058] Figure 1 is a step flow chart of a fine-grained visual language alignment contrast learning method for point cloud open vocabulary segmentation according to an embodiment of the application

[0059] Figure 2 is a step flow chart of a region-based point cloud mask according to an embodiment of the application

[0060] Figure 3 is a whole framework diagram of a fine-grained visual language alignment contrast learning method for point cloud open vocabulary segmentation according to an embodiment of the application DETAILED DESCRIPTION

[0061] The embodiments of the application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the application, and cannot be understood as a limitation of the application. For the step numbers in the following embodiments, only the setting is convenient for explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0062] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0063] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If "first" or "second" is used, it is only for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features. Furthermore, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0064] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0065] In the absence of point-by-point category annotations, and given only a region point cloud and corresponding text description, this invention aims to address the problems of discriminative information loss and coarse visual-language alignment caused by pooling operations in existing technologies during the realization of visual-language information. This invention provides a fine-grained visual-language alignment contrastive learning method for point cloud open vocabulary segmentation, effectively preserving discriminative visual information and achieving finer-grained alignment.

[0066] like Figure 1 and Figure 3 As shown, this embodiment provides a fine-grained visual language alignment and contrast learning method for point cloud open vocabulary segmentation, which may specifically include the following steps:

[0067] Step S1: Obtain raw point cloud data, which includes point cloud data of a region and corresponding text description.

[0068] In this step, the raw point cloud data is acquired. The point cloud data consists of N points P = {p1, p2, ..., p...} N A set consisting of}, where each point p n It typically includes coordinate position and RGB information, i.e., p n =P[n]=(xn ,y n ,z n ,r n ,g n ,b n The text data consists of M words W = {w1, w2, ..., w...} M A set consisting of}.

[0069] It should also be noted that in the embodiments of the present invention, [·] is used as an index operation to indicate the retrieval of the corresponding element (which can be a vector or scalar) from a set or matrix.

[0070] Step S2: Perform visual mask data augmentation based on the original point cloud to obtain visual mask data augmented point cloud data.

[0071] In this step, to reasonably remove some contextual information from the enhanced point cloud data, this embodiment of the invention introduces a region-based point cloud masking method. This method uniformly divides the scene into cubes and masks the RGB information of randomly selected points within the cubes. This embodiment of the invention uses the proposed region-based masking method to mask the enhanced point cloud data, obtaining masked point cloud data, which is then used for the masked feature reconstruction task.

[0072] See Figure 2 In some embodiments, step S2 above can be divided into the following sub-steps:

[0073] Step S21: Based on the original point cloud data and a region of a set size, obtain a series of cuboids for dividing the spatial data of the original point cloud data.

[0074] Step S22: Based on the set mask ratio and the cuboid, a series of cuboid subsets to be masked are obtained.

[0075] Step S23: Perform a masking operation on all the original point cloud data within the cuboid subset.

[0076] For example, given a region size G to represent the number of cubes, and cubes parallel to the axes in the 3D coordinate system are [(x min ,y min ,z min ),(x max ,y max ,z max The smallest cube representing the point cloud is [(0,0,0),(l,w,h)]. In this embodiment of the invention, the scene is uniformly divided into a set of cubic regions. Right now As shown below:

[0077]

[0078] x i =i·l G y j =j·w G , z k =k·h G

[0079] i, j, k ∈ {0, 1, ..., G-1}

[0080] Where, x i y j z k The points are evenly divided along the x, y, and z axes, (l G ,w G ,h G R and G represent the length, width, and height of the region, respectively. Then, this paper randomly selects R and G. 3 each region And calculate the mask flag M as follows:

[0081]

[0082] Where ∈ indicates whether a point is inside a cube. Then, the masked point cloud P is calculated as follows: m :

[0083] P m [n] = [x] n ,y n ,z n ,M[n]·r n ,M[n]·g n ,M[n]·b n ]

[0084] Step S3: Establish a point cloud segmentation model, enhance the point cloud data based on the original point cloud data and the visual mask data, and generate point-by-point feature values ​​of the original point cloud and point-by-point feature values ​​of the visual mask data-enhanced point cloud respectively using the point cloud segmentation model.

[0085] As an optional implementation, in step S3, establishing the point cloud segmentation model involves constructing a point cloud segmentation network, given a segmentation model f parameterized by θ. θ (·) For example, 3D UNet, this model can generate point-by-point feature values ​​Z = f of the original point cloud. θ (P) and point-by-point feature values ​​Z of the point cloud after mask data augmentation m =f θ (P m ),in D represents the feature dimension of the point cloud, and n represents the index value of a point.

[0086] Step S4: establishing a text feature extraction model to extract word-by-word text features and text overall features from the text description.

[0087] As an optional implementation, in step S4, the text feature extraction model is constructed by building a language model l parameterized by ω ω (·) such as Transformer, which can generate feature values for each word and a text overall feature value represented by a special character, where M is the number of words, and D is the feature dimension of the language model.

[0088] Step S5: generating a first contrastive learning loss based on the point cloud feature average value obtained from the original point cloud feature and the text overall feature.

[0089] The purpose of this step is to preliminarily align the point cloud features and the text features, so that the point cloud feature space and the text feature space are preliminarily combined, and then the point cloud model has a rough cross-modal understanding ability.

[0090] First, the point cloud region average feature is obtained:

[0091]

[0092] The point cloud average feature and the text overall feature calculate the contrastive learning loss function:

[0093]

[0094] where B represents the number of samples in a batch of data, represents the average point cloud feature from the i-th sample, represents the text overall feature from the i-th sample, and τ represents the temperature coefficient in contrastive learning. By constraining the consistency of the point cloud features and the text features from the same sample, and the inconsistency of the point cloud features and the text features from different samples, the model learns the correlation between the point cloud features and the text features.

[0095] Step S6: generating a second contrastive learning loss based on the original point cloud feature average value and the word-by-word text features.

[0096] Since multiple object concepts and attributes may appear in a text description, the text overall feature may not be able to clearly represent the appearing object concepts, and aligning the point cloud features with the text overall features may cause concept confusion.

[0097] The purpose of this step is to align the point cloud features with the more fine-grained text features.

[0098] According to the point cloud average feature and the word-by-word text feature, a similarity between the point cloud average feature and the text feature is calculated:

[0099]

[0100] wherein represents a point cloud feature average value from an i-th sample, represents a feature of a k-th word from a j-th sample.

[0101] According to the similarity between the point cloud average feature and the word-by-word text feature, a second contrastive learning loss function is calculated:

[0102]

[0103] By constraining the consistency of the similarity average value between the point cloud feature and the text feature from the same sample, the point cloud model learns the alignment between the point cloud feature and the more fine-grained word feature.

[0104] Step S7: generating a third contrastive learning loss according to the original point cloud point-by-point feature and the word-by-word text feature.

[0105] Since multiple object concepts and attributes may appear in a point cloud region, the point cloud average feature is insufficient to clearly represent the appearing objects, and the point cloud average feature may be confused with the text overall feature.

[0106] The purpose of this step is to realize the alignment between the more fine-grained point cloud feature and the text feature.

[0107] According to the point cloud average feature and the word-by-word text feature, a similarity between the point cloud average feature and the text feature is calculated:

[0108]

[0109] wherein represents a text overall feature from an i-th sample, represents a feature of a k-th point from a j-th sample.

[0110] According to the similarity between the point cloud average feature and the word-by-word text feature, a third contrastive learning loss function is calculated:

[0111]

[0112] By constraining the consistency of the similarity average value between the text overall feature and the point-by-point point cloud feature from the same sample, the point cloud model learns the alignment between the more fine-grained point cloud feature and the word feature.

[0113] Step S8: generating a consistency loss according to the original point cloud feature value and the feature value enhanced according to the visual mask data.

[0114] In order to preserve discriminative visual information, the consistency of point cloud features under different data augmentation is constrained. The feature consistency loss L consis The point-wise feature consistency is constrained in the following way:

[0115]

[0116] Where JS(·,·) is used to minimize the Jensen-Shannon (JS) divergence between different features.

[0117] Step S9: training the point cloud segmentation model according to the first contrastive learning loss, the second contrastive learning loss, the third contrastive learning loss and the consistency loss.

[0118] In this step, in order to realize fine-grained alignment of point cloud features and text features, and local consistency of visual features, and preserve context information, the first contrastive learning loss, the second contrastive learning loss, the third contrastive learning loss and the consistency loss are weighted to obtain the overall training target as follows:

[0119] L = αL CTA + βL WEA + γL PEA + δL consis

[0120] Where α is a hyperparameter for controlling the optimization strength of the first contrastive learning loss, β is a hyperparameter for controlling the optimization strength of the second contrastive learning loss, γ is a hyperparameter for controlling the optimization strength of the third contrastive learning loss, and δ is a hyperparameter for controlling the optimization of the consistency loss.

[0121] The technical solutions in the embodiments of the present application will be described below in combination with actual data:

[0122] The method provided above is applied to a commonly used point cloud segmentation dataset by the embodiments of the present application: ScanNet V2 has 20 semantic categories, including two background categories and eighteen foreground categories. The point cloud semantic segmentation model is evaluated on the validation set according to the previous method. On the ScanNet V2 dataset, the scene segmentation results are shown in Table 1 below.

[0123] Table 1 Experimental results of the embodiments on the ScanNet V2 dataset

[0124]

[0125] The embodiments also provide a fine-grained visual language alignment contrast learning implementation device, which comprises:

[0126] a data collection module configured to obtain original point cloud data, the original point cloud data comprising point cloud of a preset area and corresponding text description;

[0127] a point cloud enhancement module configured to perform visual mask data enhancement on the original point cloud data to obtain visual mask data enhanced point cloud data;

[0128] a point cloud segmentation module configured to establish a point cloud segmentation model, input the original point cloud data and the visual mask data enhanced point cloud data into the point cloud segmentation model to obtain original point cloud point-by-point feature and visual mask data enhanced point cloud point-by-point feature;

[0129] a feature extraction module configured to perform feature extraction on the text description to obtain word-by-word text feature and overall text feature;

[0130] a first loss module configured to obtain average point cloud feature according to the original point cloud point-by-point feature, and generate first contrastive learning loss according to the average point cloud feature and the overall text feature;

[0131] a second loss module configured to generate second contrastive learning loss according to the average point cloud feature and the word-by-word text feature;

[0132] a third loss module configured to generate third contrastive learning loss according to the original point cloud point-by-point feature and the word-by-word text feature;

[0133] a fourth loss module configured to generate consistency loss according to the original point cloud point-by-point feature and the visual mask data enhanced point cloud point-by-point feature;

[0134] a model training module configured to train the point cloud segmentation model according to the first contrastive learning loss, the second contrastive learning loss, the third contrastive learning loss and the consistency loss.

[0135] The fine-grained visual language alignment contrastive learning implementation device can perform the fine-grained visual language alignment contrastive learning implementation method provided by the method embodiment, and can perform any combination of the steps of the method embodiment, and has the corresponding functions and advantages of the method.

[0136] The embodiment also provides a fine-grained visual language alignment contrastive learning implementation device, which comprises:

[0137] at least one processor;

[0138] at least one memory configured to store at least one program;

[0139] When the at least one program is executed by the at least one processor, the at least one processor is caused to achieve Figure 1 the method shown.

[0140] The fine-grained visual language alignment contrast learning implementation device of the embodiment can execute the fine-grained visual language alignment contrast learning implementation method provided by the method embodiment of the application, can execute the implementation steps of any combination of the method embodiments, and has the corresponding functions and beneficial effects of the method.

[0141] The embodiment of the application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 the method shown.

[0142] The embodiment further provides a storage medium storing instructions or programs that can execute the fine-grained visual language alignment contrast learning implementation method provided by the method embodiment of the application. When the instructions or programs are executed, the implementation steps of any combination of the method embodiments can be executed, and the corresponding functions and beneficial effects of the method are possessed.

[0143] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0144] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the actual implementation of the modules, in combination with their attributes, functions, and internal relationships, are to be understood within the context of the devices disclosed herein. Thus, those skilled in the art with access to patents, scientific journals, and other public sources known by those skilled in the art will be able, using ordinary skill, to practice the application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and that the scope of the present application is to be determined by the entire scope of the claims, along with all equivalents of the claims, and that the claims are to be accorded their broadest interpretation under the patent statutes.

[0145] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer-readable medium for use by or in connection with an apparatus, method or system as described herein. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, an article of manufacture or a computer program product. The computer-readable medium can be a non-transitory computer-readable medium. The computer-readable medium can be a computer-readable non-transitory storage medium. The computer-readable medium can be a tangible computer-readable medium. The computer-readable medium can be a computer-readable tangible non-transitory medium.

[0146] Logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be embodied in non-transitory computer-readable media, which can be executed by an instruction execution system, apparatus, or device such as a computer-based system, processor, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium. The computer-readable storage medium can be a tangible computer- readable storage medium.

[0147] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0148] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above described embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, or combinations thereof, can be used with the necessary logic gates and circuitry for implementing logic functions on data signals: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0149] In the above description of the present specification, reference to the description of the terms "one embodiment", "another embodiment", or "certain embodiments" or the like means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. The illustrative appearances of the above described terms in the specification do not necessarily refer to the same embodiment or example. Moreover, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0150] Although embodiments of the present application have been shown and described, it would be recognized by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application, which is defined by the following claims and their equivalents.

[0151] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A method for implementing fine-grained visual language alignment and contrast learning, characterized in that, Includes the following steps: Acquire raw point cloud data, which includes point cloud data of a preset area and corresponding text descriptions; Visual mask data augmentation is performed on the original point cloud data to obtain visual mask data augmented point cloud data; A point cloud segmentation model is established. The original point cloud data and the visual mask data-enhanced point cloud data are input into the point cloud segmentation model to obtain the point-by-point features of the original point cloud and the point-by-point features of the visual mask data-enhanced point cloud. Feature extraction is performed on the text description to obtain word-by-word text features and overall text features; The average point cloud features are obtained based on the point-by-point features of the original point cloud, and a first contrastive learning loss is generated based on the average point cloud features and the overall text features. A second contrastive learning loss is generated based on the average point cloud features and the word-by-word text features; A third contrastive learning loss is generated based on the point-by-point features of the original point cloud and the word-by-word text features; A consistency loss is generated based on the point-by-point features of the original point cloud and the point-by-point features of the visual mask data to enhance the point cloud features. The point cloud segmentation model is trained using the first contrastive learning loss, the second contrastive learning loss, the third contrastive learning loss, and the consistency loss.

2. The fine-grained visual language alignment and contrast learning method according to claim 1, characterized in that, The step of performing visual mask data augmentation on the original point cloud data to obtain visual mask data augmented point cloud data includes: Based on the original point cloud data and a region of a set size, a series of cuboids are obtained to divide the spatial data of the original point cloud data. Based on the set mask ratio and the cuboid, a series of cuboid subsets to be masked are obtained; Perform a masking operation on all the original point cloud data within the cuboid subset.

3. The fine-grained visual language alignment and contrast learning method according to claim 1, characterized in that, First contrastive learning loss The expression is: In the formula, This indicates the number of samples in a batch of data. Indicates from the Average point cloud features of each sample Indicates from the Overall textual features of each sample This represents the temperature coefficient in comparative learning.

4. The fine-grained visual language alignment and contrast learning method according to claim 1, characterized in that, Second contrastive learning loss The expression is: In the formula, This indicates the number of samples in a batch of data. Indicates from the The average point cloud features of the sample and from the first sample The average similarity of each word-by-word text feature.

5. The fine-grained visual language alignment and contrast learning method according to claim 1, characterized in that, The third contrastive learning loss The expression is: In the formula, This indicates the number of samples in a batch of data. Indicates from the The word-by-word text features and from the first The average similarity of point-by-point features of the original point cloud.

6. The fine-grained visual language alignment and contrast learning method according to claim 1, characterized in that, The consistency loss The expression is: In the formula, This represents minimizing the divergence function between different features. Indicates the first Point cloud data, These are point-by-point features of two original point clouds and point-by-point features of a visually masked point cloud, respectively. Represents a collection of point cloud data. This indicates an index operation.

7. The fine-grained visual language alignment and contrast learning method according to claim 1, characterized in that, The expression for the loss function used to train the point cloud segmentation model is: In the formula, For the first contrastive learning loss, For the second contrastive learning loss, For the third contrastive learning loss, This results in a loss of consistency. , , , All of these are hyperparameters.

8. A fine-grained visual language alignment and contrast learning implementation device, characterized in that, include: The data acquisition module is used to acquire raw point cloud data, which includes point cloud data of a preset area and corresponding text descriptions. The point cloud enhancement module is used to perform visual mask data enhancement on the original point cloud data to obtain visual mask data-enhanced point cloud data. The point cloud segmentation module is used to establish a point cloud segmentation model. The original point cloud data and the visual mask data-enhanced point cloud data are input into the point cloud segmentation model to obtain point-by-point features of the original point cloud and point-by-point features of the visual mask data-enhanced point cloud. The feature extraction module is used to extract features from the text description to obtain word-by-word text features and overall text features; The first loss module is used to obtain average point cloud features based on the point-by-point features of the original point cloud, and to generate a first contrastive learning loss based on the average point cloud features and the overall text features. The second loss module is used to generate a second contrastive learning loss based on the average point cloud features and the word-by-word text features; The third loss module is used to generate a third contrastive learning loss based on the point-by-point features of the original point cloud and the word-by-word text features. The fourth loss module is used to generate a consistency loss based on the point-by-point features of the original point cloud and the point-by-point features of the visual mask data to enhance the point cloud. The model training module is used to train the point cloud segmentation model based on the first contrastive learning loss, the second contrastive learning loss, the third contrastive learning loss, and the consistency loss.

9. A fine-grained visual language alignment and contrast learning implementation device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7.

10. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-7.

Images