A spatial target recognition method based on bimodal fusion

Abstract

The application discloses a space target recognition method based on bimodal fusion, and belongs to the technical field of space target recognition. The method comprises the following steps: processing a space target image to generate a structured text description, and inputting the space target image and the structured text description into a visual basic model to obtain two-dimensional classification logic; performing farthest point sampling and dynamic updating on three-dimensional point cloud matched with the space target image to obtain an updated center point set of the three-dimensional point cloud; performing k-NN search, local aggregation and global pooling processing on the updated center point set to obtain global features with semantic coherence; and using the two-dimensional classification logic and the global features to obtain a final space target recognition result of the space target image to be recognized. Through cooperative processing of two-dimensional images and three-dimensional point cloud data, and in combination with field adaptation and non-parametric feature matching of a visual basic model, the application can realize high-precision and high-robustness space target recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of space target recognition, and particularly relates to a space target recognition method based on dual-mode fusion. BACKGROUND

[0002] With the deepening of human exploration and utilization of space, the importance of space situational awareness (SSA) is increasingly prominent, and the identification and classification of space targets (such as satellites, space debris, etc.) are the core links to ensure the safety of space missions and avoid orbital collisions. In recent years, deep learning technology has made breakthrough progress in image recognition due to its adaptive feature learning capability, and visual backbone networks such as convolutional neural networks (CNNs) and Transformers have been widely used in satellite identification tasks, significantly improving the recognition performance and robustness.

[0003] However, the space target recognition task faces many unique challenges, which restrict the application effect of traditional deep learning methods: the quality of space target images is inherently low due to the influence of environmental factors such as low light and background noise, and it is difficult to obtain high-quality images; at the same time, the image samples of different types of satellites have limited diversity and small inter-class differences, which makes large deep learning models prone to overfitting. Although the existing few-shot learning (FSL) method tries to solve the problem of data scarcity, it relies on the "N-way K-shot (N-class K-sample)" setting, which significantly reduces the performance when the number of satellite classes increases, and the meta-learning process has high computational cost and long time consumption. The existing public satellite data sets are mainly two-dimensional (2D) images, lacking three-dimensional (3D) geometric information. Existing multi-modal methods often fuse information through complex interaction modules, which increases the computational overhead in the inference stage and is difficult to meet the real-time requirements; at the same time, the strategy of forced modal feature alignment may suppress modal-specific information (such as geometric details of satellites), reducing the fine-grained recognition ability.

[0004] Therefore, how to fully utilize the complementary information of two-dimensional images and three-dimensional point clouds to achieve efficient and robust space target recognition in the small sample and low quality data scene has become a key problem to be solved. SUMMARY

[0005] In order to solve the above problems existing in the prior art, the application provides a spatial target recognition method based on dual-modal fusion, aiming at solving the problems of data scarcity, insufficient utilization of multi-modal information and low reasoning efficiency existing in the prior art of spatial target recognition. The method can realize high-precision and high-robustness spatial target recognition by cooperatively processing two-dimensional image and three-dimensional point cloud data, combining the field adaptation of visual basic model and non-parametric feature matching. The technical problems to be solved by the application are realized by the following technical scheme:

[0006] The application provides a spatial target recognition method based on dual-modal fusion, comprising the following steps:

[0007] S1: processing a spatial target image to be recognized to generate a structured text description, and inputting the spatial target image and the structured text description into a visual basic model to obtain two-dimensional classification logic;

[0008] S2: performing farthest point sampling on a three-dimensional point cloud matched with the spatial target image to obtain an initial center point set and dynamically updating the initial center point set to obtain an updated center point set of the three-dimensional point cloud;

[0009] S3: performing k-NN search, local aggregation and global pooling processing on the updated center point set to obtain global features with semantic coherence;

[0010] S4: using the two-dimensional classification logic and the global features to obtain a final spatial target recognition result of the spatial target image to be recognized.

[0011] Compared with the prior art, the application has the following beneficial effects:

[0012] The application provides a spatial target recognition method based on dual-modal fusion. First, a spatial target image is generated to generate a corresponding structured text description, and the structured text description and the spatial target image are input into a visual basic model. Through few-shot domain adaptation and extraction of two-dimensional image features, the effective combination of cross-modal information and the accurate adaptation of domain knowledge are realized. Then, the three-dimensional point cloud matched with the spatial target image is processed, the sampling center point is dynamically updated combined with the two-dimensional features, and the global features of the three-dimensional point cloud are obtained through a series of operations. The two-dimensional features are used to guide the optimization of the three-dimensional clustering result, and the semantic coherence and discriminability of the point cloud features are improved. Finally, the output results of the two-dimensional branch and the three-dimensional branch are fused to obtain the final recognition result, and the missing modal prediction features can also be generated when the modal is missing.

[0013] The method fully utilizes the complementary information of the two-dimensional image and the three-dimensional point cloud, solves the problems of data scarcity, insufficient utilization of multi-modal information and low reasoning efficiency in existing space target recognition, and can still realize efficient and robust space target recognition in a small sample and low-quality data scene, thereby significantly improving the recognition accuracy, robustness, reasoning efficiency and generalization ability, and being suitable for target recognition tasks in a complex space environment.

[0014] The application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of a space target recognition method based on dual-modal fusion provided by an embodiment of the application;

[0016] Figure 2 is a schematic diagram of the overall flow of two-dimensional pre-training and three-dimensional caching provided by an embodiment of the application;

[0017] Figure 3 is a test process schematic diagram of a space target recognition method based on dual-modal fusion provided by an embodiment of the application. DETAILED DESCRIPTION

[0018] In order to further illustrate the technical means and effects adopted by the application to achieve the predetermined purpose, the space target recognition method based on dual-modal fusion according to the application will be described in detail below in combination with the accompanying drawings and specific embodiments.

[0019] The foregoing and other technical contents, features and effects of the application can be clearly presented in the specific embodiment description below in combination with the accompanying drawings. Through the description of the specific embodiments, the technical means and effects adopted by the application to achieve the predetermined purpose can be understood more deeply and specifically. However, the accompanying drawings are provided for reference and illustration only, and are not intended to limit the technical solutions of the application.

[0020] It should be noted that in this document, relational terms such as first and second are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof are intended to cover non-exclusive inclusions, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0021] Embodiment one

[0022] Referring to Figure 1 , Figure 1 is a flowchart of a space target recognition method based on dual-modal fusion provided by an embodiment of the present application. The space target recognition method comprises:

[0023] S1: processing the space target image to be recognized to generate a structured text description, and inputting the space target image and the structured text description into a visual base model to obtain two-dimensional classification logic.

[0024] Step S1 of the embodiment specifically comprises:

[0025] S1.1: repeatedly inputting the space target image to be recognized into a GPT-4 model for processing to generate a plurality of corresponding structured text descriptions, so as to enhance the semantic information of the image. Specifically, the steps include:

[0026] Referring to Figure 2 , Figure 2 is a whole flowchart of two-dimensional pre-training and three-dimensional caching provided by an embodiment of the present application. The visual base model of the embodiment selects a GPT-4 model. In the experimental test process, for each type of space target, a space target image is selected, and a unified template is used: “This is a satellite called [CLASS]. Describe its look in terms of its main body, solar panels, and antenna.”, wherein [CLASS] is the target category name, which is input into the GPT-4 model as a question prompt. For each category, 10 different structured text descriptions are generated, denoted as a text set ( n is a category index, represents the n th structured text description of the category i ), and the generation process is formulated as:

[0027] ,

[0028] wherein, represents the structured text description of the category .

[0029] Further, in the actual processing of the to-be-identified space target image, the to-be-identified space target image is repeatedly input into the GPT-4 model for processing to generate corresponding multiple structured text descriptions. For example, a to-be-identified space target image is input into the GPT-4 model 10 times, and one structured text description is output each time, a total of 10 different structured text descriptions are output, so as to enhance the semantic information of the image.

[0030] In this way, with the help of the multi-modal processing capability of the GPT-4 model, rich semantic information can be extracted from the space target image and converted into structured text descriptions, laying a sufficient and valuable information foundation for subsequent analysis and processing. For example, for a certain satellite image, a text description containing details such as the main shape, the number and position of solar panels, and the type of antenna can be generated.

[0031] S1.2: input the space target image and the corresponding structured text description into the pre-trained visual base model, perform domain adaptation through the trained few-shot adapter, and obtain the two-dimensional classification logic corresponding to the space target image.

[0032] Step S1.2 of the embodiment includes the following steps:

[0033] S1.21: input the space target image and the corresponding structured text description into the pre-trained CLIP (Contrastive Language-Image Pretraining) model to obtain image features and text features, and use the image features and the text features to obtain the zero-shot classification logic of the space target image.

[0034] The visual base model used in this embodiment is a CLIP model. First, input the space target image and the corresponding structured text description into the pre-trained CLIP model to obtain image features and text features. The specific steps are as follows: input the structured text description and the corresponding space target image into the pre-trained CLIP model, and the space target image I After image encoder Processing to obtain image features , the structured text description is processed by the text encoder Processing to obtain text features , wherein, represents the to-be-identified space target image, represents the structured text description corresponding to the space target image

[0035] Subsequently, the generated image features and text features are calculated to obtain the zero-shot classification logic of the two-dimensional image (i.e., the space target image) output by the CLIP model​ , the calculation formula is:

[0036] .

[0037] S1.22: Perform few-shot data adaptation using a few-shot adapter to obtain few-shot classification logic.

[0038] First, obtain N spatial target categories and K few-shot images for each category, wherein each few-shot image contains a true value label, for all N categories and few-shot images in each category, by inputting each few-shot image into the CLIP model and the DINO model, respectively, to obtain visual features and . K Splice the visual features obtained from few-shot images of each spatial target category to obtain the static visual library corresponding to the CLIP model and the DINO model, respectively:

[0039] ,

[0040] ,

[0041] wherein and represent the static visual library corresponding to the CLIP model and the DINO model, respectively, , C represents the dimension of the feature vector of the static visual library, represents splicing.

[0042] Convert the true value labels of K few-shot images of the corresponding N spatial target categories into one-hot encoding .

[0043] Next, set a linear layer as a few-shot adapter for the CLIP model and the DINO model, respectively, with weights and , whose weights are the same as the static visual library , C wherein K here represents the dimension of the feature vector output by the linear layer, that is, equal to the dimension of the feature vector of the static visual library. The few-shot adapter as a trainable visual memory library will be optimized through image samples.

[0044] For image training samples The classification logic obtained by the few-shot adapter is obtained by the following way:

[0045] ,

[0046] ,

[0047] wherein, represents the few-shot classification logic obtained by the few-shot adapter using the CLIP model, represents the few-shot classification logic obtained by the few-shot adapter using the DINO model, as a nonlinear activation function, is a hyperparameter, preferably set to 2, i.e., , The few-shot classification logic and is learned to adapt the trainable visual memory bank to the spatial target domain.

[0048] Subsequently, by concatenating and , the bias learned by few-shot learning is obtained, and the specific formula is as follows:

[0049] .

[0050] wherein, represents the total few-shot classification logic obtained, i.e., the bias learned by few-shot learning.

[0051] S123: Fuse the zero-shot classification logic obtained by the pre-trained visual base model with the few-shot classification logic obtained by the few-shot adapter to obtain the final two-dimensional classification logic :

[0052] ,

[0053] wherein, is the zero-shot classification logic, is the few-shot classification logic, and the hyperparameter is used to control the contribution degree of the few-shot classification logic, and the weight matrix is used to determine the relative contribution of the two pre-trained visual base models (i.e., the CLIP model and the DINO model). The elements in the weight matrix are calculated by the following formula:

[0054] ,

[0055] ,

[0056] wherein, represents the few-shot classification logic and zero-shot classification logic The similarity between the few-shot classification logic and the zero-shot classification logic is calculated as follows: ; The few-shot classification logic is represented as and the zero-shot classification logic is represented as The similarity between the few-shot classification logic and the zero-shot classification logic is calculated as follows: .

[0057] In this step, by considering the zero-shot classification logic as a baseline and fusing it with the few-shot classification logic obtained by the few-shot adapter, the accuracy and robustness of the final two-dimensional classification logic can be further improved. Through a series of operations, the visual base model is effectively adapted to the spatial target domain.

[0058] It should be noted that in this embodiment, the few-shot adapter of the CLIP model and the few-shot adapter of the DINO model need to be trained in advance, and the specific training process is as follows:

[0059] For the few-shot adapter of the CLIP model and the few-shot adapter of the DINO model, the AdamW optimizer is used to jointly optimize their parameters, and the optimizer is defined as:

[0060] ,

[0061] wherein and are the parameters of the few-shot adapter of the CLIP model and the few-shot adapter of the DINO model, respectively.

[0062] During training, the initial learning rate is set to 0.0001, the training epoch is set to 100, and the numerical stability parameter is set to , represents the learning rate at the th training step, and the training sample size is in this example.

[0063] To dynamically adjust the learning rate to improve convergence, a cosine annealing scheduler is used, and the learning rate update formula is:

[0064] ;

[0065] The loss calculation uses a cross-entropy loss function:

[0066] ,

[0067] wherein is the final two-dimensional classification logic, The true label of the sample in the training process. In each iteration, the optimizer gradient is first cleared, the loss gradient with respect to the adapter parameters is calculated by backpropagation, the parameters are updated according to the current learning rate, and finally the scheduler state is updated to adjust the learning rate of the next round. Through this process, the classification error is minimized to achieve precise adaptation of the adapter to the spatial target domain data.

[0068] In the test phase, for the input spatial target test image, its corresponding two-dimensional classification logic is generated by the following steps:

[0069] The test image and its generated structured text description are jointly input into the trained and optimized pre-trained visual base model. Among them, the image extracts visual features through the visual encoder, and the structured text description extracts text features through the text encoder. First, the zero-shot classification logic is calculated, which is realized by the matrix product of the image features and the text features of the CLIP model. At the same time, the trained few-shot adapter outputs the few-shot classification logic , which integrates the adaptation results of the CLIP model and the DINO model. Finally, the two-dimensional classification logic of the test image is obtained through the fusion formula .

[0070] S2: Furthest point sampling is performed on the three-dimensional point cloud matched with the spatial target image to obtain an initial center point set and dynamically update the initial center point set to obtain an updated center point set of the three-dimensional point cloud.

[0071] The step S2 of the embodiment specifically includes:

[0072] S2.1: Furthest point sampling is performed on the three-dimensional point cloud matched with the spatial target image to obtain an initial center point set. Specifically, the following steps are included:

[0073] First, it is clear that the sampling object is the three-dimensional point cloud matched with the spatial target image to be recognized, which contains points, denoted as a point set :

[0074] ,

[0075] Each point has three-dimensional coordinate information. The sampling target is to select points from the points as the initial center point set (in this embodiment, it is set to ), and these center points need to be distributed as evenly as possible in the original three-dimensional point cloud to fully cover its geometric structure.

[0076] To obtain this initial set of center points, a point is first randomly selected from the original 3D point cloud as the initial center point, i.e., from the point set. Randomly select a point At this point, the set of center points has been selected as For the first iteration That is, the first During the process of selecting the center point, it is necessary to determine the first unselected point from the remaining unselected points. A center point. Specifically, for each unselected point... (Right now and ), calculate the relationship between this point and the set of selected center points. Find the Euclidean distances between all points, and take the minimum value among them, denoted as:

[0077] ,

[0078] in, Points that are not selected With the set of center points Points in The minimum Euclidean distance between them. Points that are not selected With the set of center points Points in The Euclidean distance between them is calculated using the following formula:

[0079] ,

[0080] in, For point The three-dimensional coordinates Point The three-dimensional coordinates.

[0081] Among all unselected points, select the ones that make The largest point is used as the new center point. ,Right now:

[0082] .

[0083] Then, the newly selected center point Add to the selected center point set ,Right now .

[0084] Repeat the above steps until completion. In the next iteration, the set of center points has been selected. This is the initial set of center points obtained.

[0085] For example, if a 3D point cloud The 128 initial center points finally obtained by the above iteration process, in which the point farthest from the current selected center point set is selected in each iteration, can be uniformly distributed in the three-dimensional space.

[0086] S2.2: Dynamically updating the initial center points in the initial center point set to obtain an updated center point set.

[0087] The step S2.2 of the embodiment specifically includes the following steps:

[0088] S2.21: Projecting the spatial target image through the visual features output by the CLIP model to the original three-dimensional point cloud to obtain the two-dimensional feature set corresponding to each initial center point in the initial center point set. , wherein, represents the initial center point corresponding to the two-dimensional feature, and the total number of initial center points.

[0089] S2.22: Performing normal vector estimation and fast point feature histogram (FPFH) calculation on the initial center point set using the Open3D tool library to extract the geometric features corresponding to the initial center points and construct a geometric feature set:

[0090] ,

[0091] wherein, represents the geometric feature set, represents the initial center point in the initial center point set corresponding to the geometric feature, the normal vector is used to represent the local orientation of the point cloud surface, and the FPFH is used to encode the geometric distribution characteristics of the local region of the point cloud, represents a real number set with a dimension of 336.

[0092] S2.23: Optimal transmission weight calculation, constructing a cost matrix by comprehensively considering the geometric distance, two-dimensional feature distance and three-dimensional coordinate distance :

[0093] ,

[0094] wherein, represents the cost of grouping any point in the three-dimensional point cloud to the initial center point , and represent the weights of the geometric feature and the two-dimensional feature, respectively, , and respectively represent the metric functions of geometric feature distance, two-dimensional feature distance and three-dimensional coordinate distance, wherein the distance metric of normal vector adopts cosine distance, the distance metric of FPFH adopts Euclidean distance, the distance metric of two-dimensional feature adopts cosine distance, and the distance metric of three-dimensional coordinate adopts Euclidean distance; and respectively represent the geometric feature and three-dimensional feature of any point in the three-dimensional point cloud, is a dynamic adjustment parameter, and its calculation formula is:

[0095] ,

[0096] wherein, is the second nearest neighbor average distance of all points in the three-dimensional point cloud, and is used to adaptively balance the weight proportion of the three-dimensional coordinate distance in the cost matrix.

[0097] S2.24: The optimal transport problem is solved by Sinkhorn algorithm iteration, that is, a weight matrix (the sum of each row is 1) is solved to minimize the cost matrix value, and the specific formula is:

[0098] ,

[0099] wherein, is the weight value in the weight matrix , and represents the dependence degree of any point on the center point .

[0100] S2.25: Based on the weight matrix , the updated center point set is obtained, and the formula is:

[0101] ,

[0102] wherein, represents the total number of points in the three-dimensional point cloud, is the updated th center point, is the th point in the three-dimensional point cloud, and the dynamic adjustment of the center point is realized by weighting average of all points according to the contribution weight, so that the updated center point set can more accurately reflect the geometric structure and semantic feature of the point cloud.

[0103] S3: The k-NN search, local aggregation and global pooling processing are performed on the updated center point set to obtain the global feature of the three-dimensional point cloud with semantic coherence.

[0104] The specific steps of step S3 of the embodiment are as follows:

[0105] S3.1: Obtain each updated centroid using k-NN search. The neighbor set of points forms the initial local clustering result of the point cloud.

[0106] Specifically, the updated center point is obtained through k-NN search. neighbor set That is, from point clouds Selected and updated center points The closest These points constitute the center point. The local neighborhood of can be expressed by the formula:

[0107] (take the previous) (the nearest point),

[0108] in, Indicates the center point The local neighborhood, i.e., the set of neighboring points. Euclidean distance is used to measure the spatial distance between points.

[0109] Similarly, the neighbor set of each updated center point is obtained, which together form the initial local clustering result of the point cloud.

[0110] S3.2: Perform point cloud trigonometric function encoding, local feature aggregation, and global feature extraction on the neighbor set of each updated center point to obtain the local aggregated features of each updated center point.

[0111] Step S3.2 of this embodiment specifically includes:

[0112] S3.21: Update the center point The neighbor set is normalized to obtain a normalized neighbor set. Then, a trigonometric function encoding method is used to calculate the position embedding of each neighbor in the normalized neighbor set, resulting in a position embedding vector for each neighbor. For points in a 3D point cloud... p The position embedding formula is:

[0113] ,

[0114] in, Point The position embedding vector, Point The three-dimensional position coordinates, For amplitude, , Indicates the fundamental wavelength, set to =1000, The number of frequency cardinalities is represented by the embedding dimension. (Setting to 72) is calculated, and the calculation formula is:

[0115] .

[0116] The position embedding obtains a position embedding vector containing the spatial position information of the points in the point cloud. Through the position embedding, the spatial position information of the points can be effectively captured, providing a basis for subsequent feature aggregation.

[0117] S3.22: The position embedding vector of each neighbor point in the neighbor point set is spliced with the position embedding vector of the corresponding center point to obtain a spliced feature vector, and the formula is:

[0118] ,

[0119] wherein, represents the spliced feature vector of each neighbor point, is the center point, represents the position embedding vector of the center point , is the neighbor point in the neighbor point set , represents the position embedding vector of the neighbor point in the neighbor point set .

[0120] S3.23: The spliced feature vector is weighted to obtain the weighted spliced feature vector of each neighbor point :

[0121] ,

[0122] wherein, represents the weighted spliced feature vector of the neighbor point in the neighbor point set , is the coordinate difference between the neighbor point and the center point, represents the embedding vector of the spatial position information of the coordinate difference between the neighbor point and the center point; Cat is a feature splicing operation; is an element product operation. Through this weighting method, the relative position information can be fully considered in the neighbor feature aggregation process, and the geometric perception ability of the feature is enhanced.

[0123] S3.24: The weighted spliced feature vector of each neighbor point is aggregated to obtain the local aggregation feature of each updated center point.

[0124] The above operation is performed for each updated center point to obtain the local aggregation feature of each center point.

[0125] ​​S3.3: After obtaining the local aggregated features of each center point, perform a global pooling operation on the local aggregated features of all center points to integrate the local aggregated features into global features that can represent the entire point cloud. The formula is as follows:

[0126] ;

[0127] in, This represents the global features of the entire 3D point cloud, and Pool represents the global pooling operation. Representing the neighbor set Neighboring points The weighted concatenated feature vector. This global feature integrates the local geometric information and overall structural features of the 3D point cloud, providing strong support for subsequent matching and recognition tasks.

[0128] In this embodiment, the methods of steps S2 and S3 are collectively named Multimodal Nonparametric Encoder (MNPE).

[0129] S4: Using two-dimensional classification logic and global features, obtain the final spatial target recognition result of the spatial target image to be identified.

[0130] Step S4 in this embodiment specifically includes:

[0131] S4.1: Obtain the training set and use step S3 to obtain the global features of each clean point cloud in the training set. Concatenate the global features of all clean point clouds into a 3D cached feature. .

[0132] Specifically, the training set used in this step includes multiple clean point clouds. Feature extraction is performed on each clean point cloud in the training set, and the feature data is cached. Global features are then extracted for each clean point cloud in the training set using step S3. Finally, the global features of all clean point clouds in the training set are... Concatenated into three-dimensional cache features :

[0133] ,

[0134] in, M This indicates the number of clean point clouds included in the training set.

[0135] Simultaneously, the class labels in the training set are converted into one-hot encodings. And cache.

[0136] S4.2: Combine the global features and 3D cached features of the 3D point cloud matched with the spatial target image. Similarity matching is performed to obtain the three-dimensional classification logic.

[0137] Specifically, please see Figure 3, Figure 3 This is a schematic diagram illustrating the testing process of a spatial target recognition method based on dual-modal fusion provided in an embodiment of the present invention. The point cloud used in the testing process is a damaged point cloud (corresponding to...). Figure 3 Given a damaged point cloud (in which case, considering factors such as local discarding, scaling, and rotation), the global feature extraction process of the test point cloud can be represented as follows:

[0138] ,

[0139] in, This represents the global features of the test point cloud.

[0140] 3D classification logic is calculated through similarity matching:

[0141] ;

[0142] in, For activation function, This represents hyperparameters, i.e. .

[0143] At the same time, step S2 is used to obtain the two-dimensional classification logic of the two-dimensional image corresponding to the test point cloud.

[0144] In practical use, the 3D point cloud corresponding to the spatial target image to be identified is usually a damaged point cloud. By processing the damaged point cloud using steps S4.1 and S4.2, the corresponding 3D classification logic can be obtained.

[0145] S4.3: The three-dimensional classification logic of the three-dimensional point cloud is fused with the two-dimensional classification logic of the spatial target image to obtain the final recognition result of the spatial target image.

[0146] Specifically, during the testing process, the three-dimensional classification logic of the three-dimensional point cloud and the two-dimensional classification logic of the spatial target image are integrated to obtain the final spatial target recognition result:

[0147] ;

[0148] in, This indicates the final spatial target recognition result. The softmax function is obtained by adjusting the... The two-dimensional classification logic of the images is averaged and normalized, then added to the three-dimensional classification logic to achieve effective fusion of multimodal information. It should be noted that in practice, K One can be chosen.

[0149] Example 2

[0150] On the basis of embodiment one, the effect of the space target recognition method based on dual-modal fusion in embodiment one is verified and illustrated through simulation experiments.

[0151] (1) Experimental conditions:

[0152] The data set of the experiment uses the NASA-2 / 3D data set, which contains 145 satellite models, of which 69 are real satellites and 76 are obtained through data enhancement technology. Each satellite model includes a corresponding 3D original model, a three-dimensional point cloud and a two-dimensional image. The two-dimensional image generation adopts a spiral view angle set, and 200 views are uniformly rendered in a specific elevation and azimuth range. A directional light is used to simulate the sun beam, and the background is set to black to simulate the space environment. The image resolution and field of view angle are fixed, and the distance between the camera and the target is dynamically calculated according to the maximum size and visible scale of the target. The three-dimensional point cloud is obtained by uniformly sampling 2048 points from the model surface using the Trimesh library. To test the robustness of the model, the ModelNet-C generated damaged test point cloud is followed.

[0153] (2) Experimental content:

[0154] For the 3D part of the space target recognition method based on dual-modal fusion proposed in the application (i.e. the part of using three-dimensional point cloud to obtain global features), the PointCLIP model and the PointCLIPV2 model in the prior art are improved in this embodiment to obtain their single sample variants (i.e. allowing them to “cache” the features of clean point clouds, which are used as the 3D part of this embodiment), so as to make a fair comparison. The hyperparameters of the PointNN model in the prior art are fine-tuned to make up for the potential shortcomings of the non-parametric method in feature matching accuracy.

[0155] (3) Experimental results and analysis:

[0156] On the NASA-2 / 3D data set, the recognition results of the 3D part under different damage types show that the method proposed in the application (corresponding to Ours-3D in Table 1) has an average accuracy of 70.78%, which is significantly higher than PointCLIP-FS (16.85%), PointCLIPV2-FS (46.19%), PointNN (37.70%) and its fine-tuned version (58.02%). Specifically, in the case of "shaking" damage to the three-dimensional point cloud, the accuracy of the method of the present application is as high as 97.93%, which is 24.27% higher than that of the comparative method; in the case of "local removal", "global addition" and "local addition" damage to the three-dimensional point cloud, the accuracy of the method of the present application is greatly improved, which reflects its strong robustness to various noise interference, which shows that the multi-modal information fusion and adaptive sampling point strategy effectively improve the accuracy and robustness of three-dimensional point cloud recognition in different damage scenarios. This effect can be clearly observed in the accuracy reduction trend presented in Table 1.

[0157] Table 1. 3D results of NASA-2 / 3D under different damage conditions

[0158]

[0159] wherein, represents the newly calibrated version of the embodiment.

[0160] Embodiment Three

[0161] On the basis of Embodiment One, another simulation experiment is carried out in this embodiment to verify and illustrate the effect of the spatial target recognition method based on dual-modal fusion described in Embodiment One.

[0162] (1) Test conditions:

[0163] In this embodiment, experiments are carried out on the ModelNet40 dataset, which contains 40 common 3D object models, covering furniture, vehicles, electronic devices and other categories. It is a classic benchmark dataset in the field of 3D point cloud recognition. The point cloud data used in the experiment is generated by uniformly sampling the model surface. A single sample point cloud contains 2048 points, and no additional corruption processing is introduced to evaluate the basic performance of the method in the general 3D object recognition scenario.

[0164] (2) Test content:

[0165] To fully verify the generalization ability of the method, the method of the present application is compared with three types of classic methods, including:

[0166] Training-based methods: including PointNet, PointNet++, PointMLP and other deep models that rely on large-scale training data for parameter learning;

[0167] Zero-shot methods: including PointCLIP, PointCLIPV2, GeoZe and other methods that do not require training set samples to participate in training, and directly realize recognition through pre-trained feature mapping;

[0168] Non-parametric method: represented by PointNN, a cache-matching paradigm, which does not rely on parameter learning.

[0169] The comparison indicators include classification accuracy, model parameter quantity and calculation efficiency (including training time or cache time), to evaluate the balance between performance and cost of the system evaluation method.

[0170] (3) Experimental results and analysis:

[0171] The experimental results are shown in Table 2. The method of the application can achieve the optimal performance in the zero-shot method and non-parametric method of the ModelNet40 dataset by deeply mining the correlation between two-dimensional visual information and three-dimensional geometric features, combined with the adaptive sampling point strategy, and the classification accuracy reaches 80.8%, which is higher than that of PointNN (79.62%) and GeoZe (70.2%) and other comparison methods.

[0172] Table 2. Results of 3D model on ModelNet40 dataset

[0173]

[0174] It should be particularly noted that the additional cache time of the method of the application is mainly due to the adaptive sampling processing of 9843 sample point clouds in the ModelNet40 dataset one by one, which involves 2D feature projection, geometric feature calculation and dynamic sampling point updating operations, and the cumulative time is about 1 hour. However, in the spatial target recognition scene, the number of available point clouds is usually only hundreds of scales (such as the NASA-2 / 3D dataset containing 145 models), at this time the cache time can be compressed to within 10 minutes, which can be ignored in practical applications. In addition, the cache-matching paradigm used in the three-dimensional point cloud processing process of the application has a significant efficiency advantage: the cache process only needs to be performed once in the model deployment stage, and the global features of the training set point clouds are stored in the feature library; then only the cosine similarity between the test point cloud and the cached feature needs to be calculated, and the single sample matching time is less than 10 milliseconds, which fully meets the real-time recognition requirement.

[0175] The above experimental results on general 3D datasets show that the method of the application not only performs well in the spatial target recognition task, but also can be effectively transferred to other 3D object recognition scenes, fully verifying its cross-domain generalization ability. This feature lays a foundation for the application of the method in downstream tasks such as automatic driving, robot navigation and industrial quality inspection, and shows a broad technical landing prospect.

[0176] In summary, the application provides a spatial target recognition method based on dual-modal fusion. First, a spatial target image is generated into a corresponding structured text description. The structured text description and the spatial target image are input into a visual base model. Through few-shot domain adaptation and two-dimensional image feature extraction, the effective combination of cross-modal information and the accurate adaptation of domain knowledge are realized. Subsequently, the three-dimensional point cloud matched with the spatial target image is processed. The two-dimensional feature is combined to dynamically update the sampling center point. Then, a series of operations are performed to obtain the global feature of the three-dimensional point cloud. The two-dimensional feature is used to guide the optimization of the three-dimensional clustering result, thereby improving the semantic coherence and discriminability of the point cloud feature. Finally, the output results of the two-dimensional branch and the three-dimensional branch are fused to obtain the final recognition result. In addition, the missing modal prediction feature can be generated when the modal is missing.

[0177] The method of the application fully utilizes the complementary information of two-dimensional images and three-dimensional point clouds, solves the problems of data scarcity, insufficient utilization of multi-modal information and low reasoning efficiency in existing spatial target recognition, and can still achieve efficient and robust spatial target recognition in small sample and low quality data scenarios. The recognition accuracy, robustness, reasoning efficiency and generalization ability are significantly improved, and the method is suitable for target recognition tasks in complex space environments.

[0178] Another embodiment of the application provides a storage medium having a computer program stored therein, wherein the computer program is used to execute the steps of the spatial target recognition method based on dual-modal fusion described in the above embodiments. In another aspect, the application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor invokes the computer program in the memory to implement the steps of the spatial target recognition method based on dual-modal fusion described in the above embodiments. Specifically, the integrated modules implemented in the form of software functional modules described above can be stored in a computer-readable storage medium. The software functional modules described above are stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method described in the embodiments of the application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0179] The above is a further detailed description of the application in combination with specific preferred embodiments, and the specific implementation of the application should not be limited to these descriptions. For ordinary skilled persons in the technical field to which the application belongs, without departing from the concept of the application, a number of simple deductions or substitutions can be made, which should be considered within the protection scope of the application.

Claims

1. A method for spatial target recognition based on bimodal fusion, characterized in that, The method comprises the following steps: S1: processing the image of the space target to be identified to generate a structured text description, and inputting the image of the space target to be identified and the structured text description into a visual base model to obtain two-dimensional classification logic; S2: performing farthest point sampling on the three-dimensional point cloud matched with the image of the space target to obtain an initial center point set and dynamically updating the initial center point set to obtain an updated center point set of the three-dimensional point cloud; S3: performing k-NN search, local aggregation and global pooling processing on the updated center point set to obtain global features with semantic coherence; S4: using the two-dimensional classification logic and the global features to obtain a final space target recognition result of the image of the space target to be identified; The S1 comprises: S1.1: inputting the image of the space target to be identified into a GPT-4 model multiple times to generate corresponding multiple structured text descriptions; S1.2: inputting the image of the space target and the corresponding structured text description into a pre-trained visual base model to perform domain adaptation through a trained few-shot adapter to obtain two-dimensional classification logic corresponding to the image of the space target; The S4 comprises: S4.1: obtaining a training set and using step S3 to obtain global features of each clean point cloud in the training set, and splicing the global features of all clean point clouds into three-dimensional cache features; S4.2: performing similarity matching on the global features of the three-dimensional point cloud matched with the image of the space target and the three-dimensional cache features to obtain three-dimensional classification logic; S4.3: fusing the three-dimensional classification logic of the three-dimensional point cloud and the two-dimensional classification logic of the image of the space target to obtain a final space target recognition result of the image of the space target.

2. The spatial target recognition method based on dual-mode fusion according to claim 1, characterized in that, The S1.2 comprises: S1.21: input the spatial target image and the corresponding structured text description into the pre-trained CLIP model to obtain the image feature and the text feature, and use the image feature and the text feature to obtain the zero-shot classification logic of the spatial target image , the calculation formula is: ， wherein, representing a spatial target image passing through an image encoder processing to obtain image features, representing a structured literal description passing through a text encoder processing to obtain text features; S1.22: using the few-shot adapter to perform few-shot data adaptation to obtain few-shot classification logic; S1.23: fusing the zero-shot classification logic obtained from the pre-trained visual base model with the few-shot classification logic obtained from the few-shot adapter, to obtain a final two-dimensional classification logic : ， wherein, represents the final obtained two-dimensional classification logic, is a zero-sample classification logic, is a few-sample classification logic, is a hyperparameter, is a weight matrix.

3. The spatial target recognition method based on dual-mode fusion according to claim 2, characterized in that, The S1.22 comprises: Obtaining a plurality of spatial target categories and a plurality of target objects in each spatial target category a plurality of few-shot images, each few-shot image containing a true value label, inputting the few-shot images into a CLIP model and a DINO model respectively to obtain corresponding visual features and respectively, and then obtaining a static visual library corresponding to the CLIP model and the DINO model respectively by splicing. ， ， wherein, and respectively represent a static visual library corresponding to the CLIP model and the DINO model, denotes splicing; corresponding N space target category K true value label of a few-shot image into one-hot encoding ; setting a linear layer as a few-shot adapter for the CLIP model and the DINO model, respectively, weights of which are and obtaining few-shot classification logic through the few-shot adapter: ， ， wherein, denotes few-shot classification logic obtained with a few-shot adapter of the CLIP model, denotes few-shot classification logic obtained with a few-shot adapter of the DINO model, is a non-linear activation function, is a hyperparameter, i.e., , ; By concatenating with the total few-shot classification logic is obtained: 。 4. The dual-modality fusion based space object identification method of claim 1, wherein, The S2 comprises: S2.1: performing farthest point sampling on the three-dimensional point cloud matched with the image of the space target to obtain an initial center point set; S2.2: dynamically updating the initial center points in the initial center point set to obtain an updated center point set.

5. The spatial target recognition method based on dual-mode fusion according to claim 4, characterized in that, The S2.2 comprises: S2.21: outputting the spatial target image through a visual feature output by a CLIP model back-projecting to the original three-dimensional point cloud to obtain each initial center point in the initial center point set a corresponding two-dimensional feature set wherein, denotes an initial center point a corresponding two-dimensional feature, denotes the total number of initial center points; S2.22: using an Open3D tool library to perform normal vector estimation and fast point feature histogram calculation on the initial center point set to extract geometric features corresponding to the initial center points and construct a geometric feature set: ， wherein, representing a set of geometric features, representing an initial center point of the set of initial center points a corresponding geometric feature; S2.23: Construct the cost matrix by comprehensively considering the geometric distance, two-dimensional feature distance and three-dimensional coordinate distance : ， wherein, denotes the cost of grouping an arbitrary point in the three-dimensional point cloud to the initial center point , and denote the weights of the geometric feature and the two-dimensional feature, respectively, , and denote the metric functions of the geometric feature distance, the two-dimensional feature distance and the three-dimensional coordinate distance, respectively, and denote the geometric feature and the three-dimensional feature of an arbitrary point in the three-dimensional point cloud , respectively, is a dynamic adjustment parameter; S2.24: iteratively solving an optimal transport problem to minimize the cost matrix value through a Sinkhorn algorithm, and the specific formula is: ， wherein, is a weight matrix, is a weight matrix the weight values in the weight matrix, indicating the dependence degree of any point on the center point , denotes the total number of points in the original point cloud; S2.25: Based on the weight matrix Computing the updated center point set ， in, This represents the total number of points in a 3D point cloud. For the updated number One central point, For the first in a 3D point cloud One point.

6. The spatial target recognition method based on dual-mode fusion according to claim 5, characterized in that, The S3 comprises: S3.1: Obtain each updated centroid using k-NN search. The neighbor set of points forms the initial local clustering result of the point cloud; S3.2: performing point cloud trigonometric function coding, local feature aggregation and global feature extraction on the neighbor point set of each updated center point to obtain local aggregation features of each updated center point; S3.3: after obtaining the local aggregation features of each center point, performing a global pooling operation on the local aggregation features of all center points to integrate the local aggregation features into global features capable of representing the entire point cloud: ， wherein, represents the global feature of the entire three-dimensional point cloud, Pool is a global pooling operation, represents the neighboring point set the neighboring point in the weighted spliced feature vector.

7. The spatial target recognition method based on dual-mode fusion according to claim 6, characterized in that, The S3.2 comprises: S3.21: Normalizing the neighbor point set of the updated center point obtaining a normalized neighbor point set by normalizing the neighbor point set of the updated center point, and calculating a position embedding of each neighbor point in the normalized neighbor point set by using a trigonometric function encoding mode to obtain a position embedding vector of each neighbor point in the normalized neighbor point set. ， wherein a position embedding vector for a point three-dimensional position coordinates for a point an amplitude , a base wavelength a number of frequency bases;​​ S3.22: Concatenate the position embedding vector of each neighbor in the neighbor set with the position embedding vector of the corresponding center point to obtain a concatenated feature vector: ， wherein, is a set of neighboring points, is a neighboring point in the set of neighboring points, denotes a feature vector of each neighboring point after concatenation, is a center point, denotes a position embedding vector of the center point , and denotes a position embedding vector of a neighboring point in the set of neighboring points , and denotes a position embedding vector of a neighboring point in the set of neighboring points S3.23: Weighting processing is performed on the spliced feature vectors to obtain each neighbor point Weighted spliced feature vectors: ， wherein, denotes neighboring points of the set of neighboring points denotes a neighboring point in the set of neighboring points a weighted concatenation feature vector, is a coordinate difference of the neighboring point and the center point is an element-wise multiplication operation, denotes an embedding vector of spatial position information of the coordinate difference of the neighboring point and the center point; S3.24: For each neighbor point The weighted stitching feature vectors are aggregated to obtain the local aggregated features of each updated center point.

Images