Domain knowledge enhancement methods and devices for large-scale infrastructure defect detection models

Abstract

This invention provides a domain knowledge enhancement method and device for large-scale infrastructure defect detection models. First, a multimodal dataset in the infrastructure defect detection domain is constructed, and a knowledge base containing defect types and background information is built and transformed into a knowledge graph. Based on this graph, a high-dimensional feature vector incorporating domain knowledge is generated using a graph neural network. Simultaneously, single-modal features are extracted from multimodal data such as images, point clouds, and depth maps, and fused using a cross-attention mechanism to obtain multimodal fused features. Then, the domain knowledge vector and the multimodal fused vector are concatenated and input into the large-scale model for training, achieving high-precision detection of infrastructure defects. This invention, by introducing structured domain knowledge and a multimodal fusion mechanism, effectively improves the defect detection accuracy and robustness of large-scale models in complex infrastructure scenarios, demonstrating good engineering applicability and promotional value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and specifically to a domain knowledge enhancement method for large-scale infrastructure defect detection models. Background Technology

[0002] Defect detection in infrastructure is a crucial step in engineering construction and maintenance, directly impacting public safety and facility lifespan. With the expansion of infrastructure construction and the increase in its service life, defect detection of various facilities has become a vital link in ensuring safe operation. However, traditional defect detection methods rely on human experience, resulting in low efficiency and high costs. In recent years, large-scale deep learning-based models have made significant progress in areas such as image processing and point cloud analysis, demonstrating powerful feature extraction capabilities and defect detection potential. However, the performance of these data-driven models is often limited by the scale and quality of the training data, especially in domain-specific defect detection scenarios, where they are prone to poor generalization and insufficient detection accuracy.

[0003] Domain knowledge, as a summary of experience accumulated by human beings over a long period of practice, includes information such as materials used in infrastructure construction, typical defect types and patterns, operating environment conditions of equipment, and industry standards and norms. It can effectively make up for the shortcomings of data-driven models and provide additional background information to improve model performance.

[0004] Knowledge graphs are a technology that represents knowledge in a structured way. Through a graph structure of nodes and relationships, they clearly describe entities and the relationships between them. Domain knowledge is the main source of knowledge graphs; for example, expert knowledge and historical fault data can be transformed into graph nodes and relationships. By incorporating domain knowledge into knowledge graphs, the problem of insufficient data can be effectively compensated for, the model's detection capabilities in specific scenarios can be improved, and the relevant inferences from domain knowledge can be introduced into the model's reasoning process, enhancing the credibility of the model's output results.

[0005] Graph neural networks have a natural advantage in processing graph-structured data, enabling them to efficiently mine semantic information and structural characteristics from knowledge graphs. By aggregating information from neighboring nodes and updating the feature representations of nodes, they can generate high-dimensional feature vectors containing semantic information, providing accurate domain knowledge representations for defect detection.

[0006] Multimodal data fusion refers to inputting data of different types into a large-scale infrastructure defect detection model. By fusing image and point cloud data, the texture and three-dimensional morphological features of defects can be captured simultaneously, enabling comprehensive detection of cracks, corrosion, and spalling. Combining depth maps with environmental sensor data can improve the sensitivity to potential defects caused by environmental factors (such as humidity and temperature), significantly enhancing the adaptability of the defect detection model in complex scenarios.

[0007] In summary, the introduction of domain knowledge, the fusion of multimodal data, and the application of graph neural networks provide strong technical support for large-scale infrastructure defect detection models. Designing a domain knowledge enhancement method based on knowledge graphs and multimodal data fusion will provide an efficient, accurate, and robust solution for infrastructure defect detection, while also opening up new pathways for intelligent infrastructure operation and maintenance. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention proposes a domain knowledge enhancement method and device based on multimodal data fusion. This improves the detection accuracy and robustness of large-scale vertical defect detection models, particularly for defect detection in the infrastructure construction field. Its input includes data of various infrastructure types such as defect images, depth images, and point clouds, and its output is the labeled defect detection results corresponding to the input data types.

[0009] The domain knowledge enhancement method for a large-scale infrastructure defect detection model described in this invention includes the following steps:

[0010] Construct a multimodal dataset for infrastructure defect detection;

[0011] Using the aforementioned multimodal dataset, a domain knowledge base for infrastructure defect detection is constructed, including infrastructure knowledge background and defect types;

[0012] A knowledge graph is constructed based on the domain knowledge base, mapping the entities and relations in the domain knowledge base to nodes and relations in the knowledge graph;

[0013] Based on the knowledge graph, a high-dimensional feature vector containing domain knowledge is generated using a graph neural network.

[0014] Single-modal features are extracted for each modality in the multimodal data, and then a multimodal fusion feature vector is generated by fusing the multimodal data.

[0015] The high-dimensional feature vector is concatenated with the multimodal fused feature vector and input into a large model for training;

[0016] Use a well-trained large model to detect infrastructure defects.

[0017] Furthermore, the constructed infrastructure defect detection domain dataset includes:

[0018] Collect multimodal data, including RGB images, depth images, point cloud data, and sensor data;

[0019] The data is labeled with truth values, including defect type, location, attributes, and environmental information;

[0020] The data is divided into training, validation, and test sets.

[0021] Furthermore, the construction of the infrastructure defect detection domain knowledge base includes:

[0022] Collect literature, industry standards, and expert experience related to infrastructure deficiencies;

[0023] Defect types are classified according to infrastructure category and defect nature;

[0024] The categorized defect data is preprocessed into a structured format and stored as form-based data.

[0025] Furthermore, the construction of the knowledge graph includes:

[0026] Define infrastructure type nodes, defect type nodes, and environmental condition nodes;

[0027] Define the relationship type between nodes;

[0028] The structured information in the domain knowledge base is mapped to nodes and edges in the knowledge graph.

[0029] Furthermore, the generation of high-dimensional feature vectors using graph neural networks includes:

[0030] Initialize attribute vectors for nodes in the knowledge graph;

[0031] Graph neural networks are used to aggregate neighbor node information and update node feature representations layer by layer.

[0032] Generate high-dimensional feature vectors that incorporate domain knowledge.

[0033] Furthermore, the multimodal data fusion includes:

[0034] Single-modal features were extracted from RGB images, depth maps, and point cloud data, respectively.

[0035] Standardize the single-modal features;

[0036] Cross-attention mechanism is used to capture intermodal correlations, and features are fused through a weighted strategy.

[0037] Preferably, the large model is CLIP or LLaMA, and the training of some weights of the large model is frozen.

[0038] Furthermore, the method also includes optimizing the large model parameters using a cross-entropy loss function.

[0039] Based on the same inventive concept, the present invention also discloses an electronic device, comprising:

[0040] One or more processors;

[0041] Storage device for storing one or more programs;

[0042] When one or more programs are executed by the one or more processors, the one or more processors implement a domain knowledge enhancement method for a large model of infrastructure defect detection.

[0043] Based on the same inventive concept, the present invention also designs a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements a domain knowledge enhancement method for a large-scale infrastructure defect detection model.

[0044] Advantages of this invention:

[0045] 1. This invention innovatively incorporates domain knowledge into a large-scale defect detection model. Specifically, by constructing a domain knowledge base for infrastructure defect detection, it structures and maps professional knowledge such as material information, typical defect types, equipment operating conditions, and industry standards onto a knowledge graph, thereby compensating for the shortcomings of traditional data-driven models in defect detection. The integration of domain knowledge provides the model with additional background information, effectively improving the model's accuracy in identifying different types of infrastructure defects even in the absence of large amounts of labeled data, and enhancing detection accuracy through knowledge reasoning.

[0046] 2. This invention innovatively integrates multiple modal information, such as defect images, depth images, and point cloud data, into a large-scale defect detection model, comprehensively capturing the texture and three-dimensional morphological features of defects. This enables the model to analyze defects from different dimensions, improving the detection capability for defect types such as cracks, corrosion, and spalling in complex structures and infrastructure under various working conditions. It also provides greater robustness when facing diverse defects in different types of infrastructure, reducing the risk of false positives and false negatives.

[0047] 3. This invention innovatively proposes a multimodal data fusion technology that integrates different types of data such as images, depth maps, and point clouds, and establishes relationships between different modalities through a cross-attention mechanism. This method can comprehensively consider different features of defects, avoid the limitations of single-modal data, improve the comprehensive detection capability for multiple defect types, and provide more comprehensive and accurate defect identification results. Attached Figure Description

[0048] Figure 1 This is a framework diagram of the domain knowledge enhancement model for the large-scale infrastructure defect detection model of the present invention.

[0049] Figure 2 This is a diagram of the multimodal feature fusion vector framework in this invention. Detailed Implementation

[0050] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0051] Example 1

[0052] This embodiment discloses a domain knowledge enhancement method for a large-scale infrastructure defect detection model, including the following steps:

[0053] Step S1 involves constructing a multimodal dataset for infrastructure defect detection, used for training large-scale vertical models in infrastructure defect detection. Multimodal data acquisition is used to construct datasets representing potential defect types in infrastructure such as bridges, tunnels, and power cables, including cracks, corrosion, and spalling. The specific implementation process is as follows:

[0054] Step S1-1: Determine data requirements based on the defect type and detection scenario of the target infrastructure, including multimodal information such as RGB images, point clouds, depth images, and sensor data, and perform data acquisition. Use a high-resolution camera or drone camera to photograph the surface of the infrastructure, capturing surface defects such as cracks, peeling, and corrosion; use a depth camera to obtain object distance information, reflecting the depth changes of the defect area.

[0055] Steps S1-2 involve using specialized tools to perform ground truth annotation on defect types, locations, attributes, and environmental information. In this embodiment, the Labelme annotation tool is used to perform ground truth annotation on the collected multimodal data, annotating each sample with the defect type (cracks, corrosion, spalling, etc.), specific location (determined by selecting a box or marking coordinates to define the defect area), various defect attributes (size, shape, severity, etc.), and environmental information at the time of collection (light, weather, temperature, humidity, etc.). After annotation, domain experts review the annotation results to ensure the accuracy and consistency of the data annotation. Subsequently, data cleaning, enhancement, and modal alignment operations are performed to unify the data format and improve sample diversity.

[0056] Steps S1-3 divide the data into training, validation, and test sets according to a set ratio to ensure data diversity and generalization ability.

[0057] Step S2: Construct a domain knowledge base for infrastructure defect detection, summarizing the knowledge background and defect types of infrastructure. The specific process of building the infrastructure defect detection domain knowledge base is as follows:

[0058] Step S2-1 involves collecting key technical data, common defect types and their formation mechanisms, and detection methods related to infrastructure defect detection from existing academic literature, industry standards, technical reports, and other materials. This includes the basic structure of different types of infrastructure, common defect types, defect formation mechanisms, and various detection methods. Information on the frequency of occurrence, impact range, and failure mechanisms of common defects is also collected.

[0059] Step S2-2: Based on the raw data collected in Step S2-1, infrastructure defects are further subdivided into defect types, categorized according to different dimensions, such as bridge cracks, tunnel spalling, and cable insulation aging. Defects are also classified according to their nature into physical defects, chemical defects, and mechanical defects. For example, bridge defects are categorized as "bridge cracks" and "bridge spalling," while tunnel defects are categorized as "tunnel spalling" and "tunnel leakage." Similarly, cable defects can be subdivided into "cable aging" and "cable breakage." Further, defects are classified according to their nature into physical defects (such as cracks and deformation), chemical defects (such as corrosion and oxidation), and mechanical defects (such as fatigue and fracture). A taxonomic approach is used to organize structured knowledge data tags, providing a unified tagging system for the subsequent construction of a domain knowledge database.

[0060] Step S2-3 involves preprocessing the data collected in Step S2-2 by removing duplicate records, correcting errors, and supplementing missing information, as domain knowledge often suffers from inconsistent formats and redundant information. Simultaneously, various data formats such as text, tables, and images are converted into a unified structured format, and images and point clouds are labeled as tag files containing defect types and locations. This unified conversion of all data types into a structured format involves transforming knowledge base information into CSV files and converting tag files containing attributes such as defect type, location, and size into JSON format.

[0061] S2-4, after completing data preprocessing, stores information such as defect classification, attributes, and detection methods as tabular data, and generates a corresponding identifier for each record, thus completing the construction of the infrastructure defect detection domain knowledge base. A data record R within the domain knowledge base is represented as... , where t represents the type of infrastructure (such as bridges, tunnels, etc.). a n It represents defect attributes (such as crack width, material strength, etc.).

[0062] Step S3: Based on the infrastructure defect detection domain knowledge base constructed in step S2, it is mapped to nodes and relationships in a knowledge graph, thus realizing the construction of a knowledge graph based on domain knowledge. The specific process of constructing a knowledge graph based on infrastructure defect detection domain knowledge is as follows:

[0063] Step S3-1: Define the overall structure of the knowledge graph based on the characteristics of domain knowledge. This includes defining the infrastructure types. (e.g., bridges, tunnels), defect types (e.g., cracks, peeling) and environmental conditions Parameters such as humidity and temperature are defined as node types. Simultaneously, the types of relationships between nodes and the causal relationships between environmental condition nodes and defects are clarified. For example, "causing" (humidity causes corrosion); the dependency relationship between defect types, nodes, and infrastructure. "Belongs to" (cracks belong to bridge defects); the relationship of defects occurring in a specific environment or location. Relationships such as "occurred" (stripping occurred on the tunnel wall) are defined; and a directed graph is defined. , where the set of nodes Edge set It is composed of different relationships.

[0064]

[0065] Furthermore, the structured domain knowledge compiled in step S2 is mapped to the graph structure of the knowledge graph in step S3-1, and the attribute values ​​of each node are further initialized to give it richer semantic information. The infrastructure type and defect attributes in each record are defined as nodes in the knowledge graph as follows:

[0066]

[0067] in, Indicates the type of infrastructure node. This represents a defect attribute node.

[0068] The relationships between infrastructure types and attributes in the records are mapped to edges in the knowledge graph, defined as follows:

[0069]

[0070] in, V It is a set of nodes, representing the infrastructure type, defect type, and their attributes; E It is a set of edges, representing the relationships between nodes.

[0071] Step S3-3: Initialize attribute values ​​for each node in the knowledge graph. For example, a defect type node may include attributes such as shape, size, and frequency of occurrence; an environmental condition node may include characteristics such as humidity, temperature, and wind speed. These attribute values ​​can be extracted directly from the domain knowledge base and then subjected to data estimation and normalization processing according to actual needs.

[0072] Step S4: Construct a domain knowledge network based on a graph neural network. The entities and relationships in the knowledge graph constructed in Step S3 are used to build nodes and edges, forming a graph structure. Attributes of nodes, such as defect type and environmental conditions, are initialized as feature vectors. The graph neural network is used to update the feature representation of nodes layer by layer by aggregating information from neighboring nodes and its own features. After multi-layer feature propagation, a high-dimensional feature vector is generated for each node, containing structured and semantic information from the infrastructure defect detection domain. Specifically, this includes the following steps:

[0073] Step S4-1: Based on the knowledge graph constructed in step S3, add edge weight information to enhance the graph structure's ability to express the strength of relationships between nodes. Define a weighted directed graph structure. ,in It is an adjacency matrix. Represents the weight of an edge, measuring the value of a node. and The importance of the relationships between nodes is emphasized. In this embodiment, edge weights are assigned different weights to different types of edges based on expert experience. Specifically, the higher the relevance of graph nodes, the higher the edge weight value, thereby improving the feature learning ability of the graph neural network.

[0074] Step S4-2: In the input phase of the graph neural network model, each node is assigned a numerical feature vector, and feature propagation and learning are performed. Define the node feature matrix. ,in This is the dimension of the node features. Initial feature matrix of the nodes. With weighted directed graph structure The set of nodes V corresponds to each node, and the feature vector of each node is generated by its attributes:

[0075]

[0076] Step S4-3: Select a graph neural network model, such as GCN (Graph Convolutional Network) or GraphSAGE (Graph Sample and Aggregate), and define its key parameters, such as the number of network layers L, the number of hidden units H, and the activation function. Node features capture global and local information through neighbor aggregation and self-updating. The aggregation function can be average pooling, max pooling, or attention mechanisms, for example, using attention mechanisms to assign different weights to neighbor nodes to highlight the contributions of important relational nodes.

[0077] In this embodiment, a GAT (Graph Attention Networks) model is selected for feature learning, and the network has three layers to ensure sufficient information aggregation capability. The hidden layer has a 128-dimensional dimension to store the intermediate representations of nodes. The ReLU activation function is used to increase the non-linear expressive power of the features. During the computation of the graph neural network, a neighbor feature aggregation mechanism is used to update the representation of each node.

[0078] Step S4-4: The final feature representation of each node is in the last layer of the graph neural network. Fixed to a 128-dimensional high-dimensional vector space, the final feature representation of each node includes both the structural features of infrastructure defects and the implicit information in the graph structure.

[0079] Step S5 involves constructing a multimodal fusion feature vector. Deep learning-based network models (such as CNN and Transformer) are used to extract features from each input modality of data. These features are then comprehensively represented in a unified feature space. Each modality's feature vector is processed independently to generate a single-modality intermediate representation. A cross-attention module is then used to establish relationships between different modalities. Finally, a weighted ensemble strategy is employed to fuse the prediction results from each modality, achieving multimodal data fusion. This comprehensively captures defect characteristics from a multimodal perspective, resulting in a multimodal fusion feature vector. The details are as follows:

[0080] Step S5-1, Given multimodal input data , D img , D depth , D point These represent RGB images, depth maps, and point cloud data, respectively. For each modality of data... ,use Figure 2 A dedicated encoder model was used to extract single-modal features. ResNet101 was used for image feature extraction to obtain a 512-dimensional feature vector. A convolutional neural network 3D-CNN was used to extract spatial features of the depth map to obtain a 256-dimensional feature vector. PointNet was used to extract global features of the point cloud data to obtain a 1024-dimensional feature vector.

[0081] Step S5-2, for each single-modal feature Standardization is performed to ensure that the mean of the eigenvalues ​​is 0 and the variance is 1, thus eliminating differences in the range of eigenvalues ​​between modes. The standardization formula is as follows:

[0082]

[0083]

[0084]

[0085] in, It is modal eigenvectors, , They are respectively The mean and standard deviation, It is an eigenvector Dimensions yes Standardized feature vectors yes The specific value of the j-th dimension.

[0086] Step S5-3: Use a cross-attention mechanism to capture the correlation between different modalities, and fuse the updated modal features through a weighted strategy, using weights... Manually adjust the importance of different modalities to obtain a multimodal fusion feature vector. For any two modes , Calculate attention weights and the feature representation after interaction :

[0087]

[0088]

[0089] in, , , To obtain the query, key, and value vectors of the feature vector through linear transformation, , , It is a learnable weight matrix. It is a scaling factor used to stabilize the gradient.

[0090] The updated modal features are fused using a weighted strategy, with weights applied. Adjusting the importance of different modes:

[0091]

[0092] in, .

[0093] Finally, the multimodal fusion feature vector is obtained. .

[0094] Step S6: The high-dimensional feature vector of infrastructure domain knowledge generated in step S4 is... and the multimodal fusion feature vector generated in step S5 By concatenating the features along the dimensional lines, a feature vector is obtained that incorporates domain knowledge and multimodal data fusion. .

[0095] Step S7: Model building. The feature vectors obtained in step S6 are input into a large model (such as CLIP, LLaMA, etc.) as the basic framework, and the weights of the large model are frozen during training to construct a large model for infrastructure defect detection, realizing the training of a domain knowledge-enhanced infrastructure defect detection large model. The specific process is as follows:

[0096] Step S7-1: In this embodiment of the invention, the general multimodal large model CLIP is selected as the basic framework, and ViT-B / 32 is selected as the feature encoder.

[0097] Step S7-2, concatenate the feature vectors Input to the selected large model In this process, the weights of the large model are frozen during training to construct a large model for infrastructure defect detection, thereby enabling the training of a domain knowledge-enhanced infrastructure defect detection large model.

[0098] Step S7-3: Calculate the model prediction results using the cross-entropy loss function. and real labels Error between:

[0099]

[0100] in, It refers to the batch size. It is the number of category labels. It is a sample Real labels in categories The label value (usually 0 or 1). The model predicts the sample. Category The probability of.

[0101] Update model parameters by minimizing cross-entropy loss. :

[0102]

[0103] in, It's the learning rate. It is the gradient of the loss function with respect to the model parameters.

[0104] The trained model It can perform high-precision defect detection on input modal data such as images or point clouds, and output prediction results:

[0105]

[0106] in, The defect prediction results output by the model.

[0107] Example 2

[0108] Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in Embodiment 1.

[0109] Since the device described in Embodiment 2 of this invention is the electronic device used in implementing the domain knowledge enhancement method for large-scale infrastructure defect detection in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0110] Example 3

[0111] Based on the same inventive concept, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0112] Since the device described in Embodiment 3 of this invention is a computer-readable medium used to implement the domain knowledge enhancement method for large-scale infrastructure defect detection in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0113] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A domain knowledge enhancement method for infrastructure defect detection large models, applied to defect detection of bridges, tunnels or power cable facilities, characterized in that, The method comprises the following steps: constructing a multi-modal dataset in the field of infrastructure defect detection; constructing a domain knowledge base for infrastructure defect detection using the multi-modal dataset, the domain knowledge base comprising infrastructure type knowledge, defect type knowledge, and environmental condition knowledge; constructing a knowledge graph based on the domain knowledge base, wherein infrastructure type nodes, defect type nodes, and environmental condition nodes are defined, and edge types including causal correlation edges and subordinate correlation edges are defined, the causal correlation edges being used to represent the causal relationship between the environmental condition nodes and the defect type nodes; constructing the knowledge graph as a weighted directed graph structure, and assigning preset weights to different types of edges based on expert experience; generating a high-dimensional feature vector containing domain knowledge using a graph neural network based on the weighted directed graph structure; extracting single-modal features for each modality in the multi-modal data, the multi-modal data comprising RGB images, depth images, and point cloud data; using a cross-attention mechanism to capture inter-modal correlation and fuse features of the single-modal features of the depth images and the point cloud data, and generating a multi-modal fusion feature vector; concatenating the high-dimensional feature vector and the multi-modal fusion feature vector, and inputting them into a CLIP or LLaMA large model with some frozen weights for training; using the trained large model to perform infrastructure defect detection.

2. The method of claim 1, wherein, The construction of the dataset in the field of infrastructure defect detection comprises: collecting multi-modal data, including RGB images, depth images, point cloud data, and sensor data; annotating the data with true values, including defect type, location, attribute, and environmental information; dividing the data into a training set, a validation set, and a test set.

3. The method of claim 1, wherein, The construction of the domain knowledge base for infrastructure defect detection comprises: collecting literature, industry standards, and expert experience related to infrastructure defects; classifying defect types according to infrastructure categories and defect properties; preprocessing the classified defect data into a structured format and storing it as tabular data.

4. The method of claim 1, wherein: The graph neural network is a graph attention network or a relational graph convolution network.

5. The method of claim 1, wherein: The generation of the high-dimensional feature vector using the graph neural network comprises: initializing attribute vectors for the nodes in the knowledge graph; using the graph neural network to aggregate neighbor node information according to the preset weights and update node feature representations layer by layer; generating a high-dimensional feature vector containing domain knowledge.

6. The method of claim 2, wherein, The feature fusion using the cross-attention mechanism comprises: extracting single-modal features of RGB images, depth images, and point cloud data respectively; standardizing the single-modal features; using the depth image features or the point cloud features as query vectors, and other modal features as key and value vectors, to calculate cross-attention weights; based on the cross-attention weights, generating the multi-modal fusion feature vector through a weighted fusion strategy.

7. The method of claim 1, wherein: The method further comprises optimizing the parameters of the large model using a cross-entropy loss function.

8. An electronic device, comprising: comprise: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

9. A computer readable medium having stored thereon a computer program, characterized in that: The program, when executed by the processor, implements the method of any one of claims 1-7.

Images