Cross-modal coupling method and system for semiconductor multi-modal data

Abstract

The invention belongs to the field of autonomous high-throughput characterization of semiconductor materials, and discloses a cross-modal coupling method and system for semiconductor multi-modal data, and the method comprises the steps: firstly, respectively extracting the features of an image, an electricity and a process modal through a multi-modal perception encoder; secondly, reconstructing missing features based on heterogeneous data relevance by using a modal missing compensation module constructed by a graph neural network; dynamic interaction and fusion among the features are realized through a multi-modal cross attention mechanism; inputting the fusion features into a multi-scale output module to generate point location-level, block-level and wafer-level features; and finally, optimizing cross-modal coupling by adopting progressive three-stage training. By adopting the method, material and device characteristics can be comprehensively modeled, cross-modal potential association can be deeply captured, data missing can be effectively compensated, structural adaptation of a semiconductor specific task can be realized, and fusion expression capability and robustness can be remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous high-throughput characterization technology of semiconductor materials, specifically relating to a cross-modal coupling method and system for semiconductor multimodal data. Background Technology

[0002] Currently, third-generation semiconductor materials (such as gallium nitride GaN, silicon carbide SiC, and gallium oxide Ga2O3) are showing broad application prospects in key fields such as optoelectronic devices, high-frequency communications, and power devices. To ensure the performance stability and process controllability in their practical applications, it is necessary to collect multiple modal data, including surface and structure images (AFM, SEM, EBSD), electrical parameters (resistivity, mobility, IV / CV curves), and growth or process parameters (temperature, pressure, time, gas flow rate, etc.), during material research and development, device manufacturing, and wafer inspection. These data originate from different devices, sampling times, and physical dimensions, exhibiting characteristics of "modal heterogeneity, inconsistent dimensions, asynchronous timing, and large differences in acquisition granularity." Furthermore, they are interconnected (e.g., surface morphology distortion is often accompanied by spectral redshift, and electrical performance degradation often stems from structural defects). Single-modal data cannot fully reflect the true state of materials or devices. Therefore, there is an urgent need for data fusion and feature extraction technologies for semiconductor multimodal scenarios to achieve structured fusion expression, possess strong interpretability and robust compensation mechanisms for modal loss, and adapt to the needs of multi-scale and multi-task subsequent processing, providing support for material modeling, data analysis, and industrial decision-making.

[0003] Existing semiconductor-related AI models mostly employ a single-modal approach, making it difficult to comprehensively model material and device characteristics and failing to meet the need to judge semiconductor material properties through multiple indicators. Current mainstream methods for multimodal fusion (such as "feature stitching" and "modal concatenation input") fail to deeply model the contextual relationships and coupling characteristics between different modalities, making it difficult to capture potential semantic associations across modalities and limiting fusion effects and expressive capabilities. In actual semiconductor testing scenarios, when some modal data is missing or incomplete due to equipment maintenance, testing conditions, or data loss, existing fusion algorithms often rely on full-modal input, resulting in a significant performance degradation and a lack of effective compensation or reconstruction mechanisms. Furthermore, semiconductor data exhibits varying sampling frequencies and large dimensional spans (images are 2D, processes are structured tables), with different modalities having inconsistent scales in time, space, and semantics. Existing methods are mostly general model transfer applications, lacking structural and task adaptation capabilities specific to semiconductor scenarios.

[0004] In summary, the current third-generation semiconductor field faces several technical challenges in multimodal data processing, including the inability of single-modal models to reflect the true state of materials and devices, the lack of in-depth modeling of modal correlations in multimodal fusion methods, insufficient robustness of algorithms when data is missing, and the lack of adaptability of general models to semiconductor scenarios. Summary of the Invention

[0005] This invention provides a cross-modal coupling method and system for semiconductor multimodal data. This method can solve the problems that single-modal models cannot reflect the real state of materials and devices, multimodal fusion methods do not deeply model modal correlations, algorithms are not robust enough when data is missing, and general models lack semiconductor scene adaptability.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A cross-modal coupling method for semiconductor multimodal data includes: Multimodal data of semiconductor samples are acquired and each modal data is preprocessed separately; the multimodal data includes image modal data, electrical modal data and process modal data; A pre-built multimodal sensing encoder network is used to extract features from each modal data to obtain image modal features, electrical modal features, and process modal features; Image modal features, electrical modal features, and process modal features are input into a pre-constructed modal loss compensation module to predict and obtain reconstructed modal features; the modal compensation module is constructed based on a graph neural network. A multimodal cross-attention module is used to interact and fuse image modal features, electrical modal features, process modal features, and remodeling modal features to output a fused feature vector; The fused feature vector is input to the multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, tile-level output, and wafer-level output; A progressive three-stage training approach is adopted to jointly optimize multi-scale fusion features for multi-modal data, so as to achieve cross-modal coupling of semiconductor multi-modal data.

[0007] Furthermore, the acquisition of multimodal data from the semiconductor sample and the preprocessing of each modal data include: Image modal data, electrical modal data, and process modal data were separately acquired from the semiconductor sample. The image modal data included AFM and SEM grayscale images; the electrical modal data included surface potential, mobility, and resistivity; and the process modal data included temperature, pressure, reaction time, and gas ratio structured process parameters. The image modal data is scaled to a uniform preset size using bilinear interpolation and then normalized after Gaussian filtering. Electrical modal data and process modal data were standardized using Z-score, and then... Criteria for removing outliers.

[0008] Furthermore, the pre-constructed multimodal sensing encoder network extracts features from each modal data to obtain image modal features, electrical modal features, and process modal features, including: For each modal data, a corresponding feature extraction sub-network is constructed to obtain the multimodal sensing encoder network; among them, the image modal data adopts the Swing Transformer encoder; the electrical modal data and the process modal data both adopt the multilayer perceptron. Image modal data is input into the Swing Transformer encoder, and image modal features are output through the convolutional block attention module; Electrical modal data and process modal data are input into a multilayer perceptron, which outputs electrical modal features and process modal features.

[0009] Furthermore, the step of inputting image modal features, electrical modal features, and process modal features into a pre-constructed modal loss compensation module to predict and obtain reconstructed modal features includes: A modality missing compensation module is constructed; wherein, the modality missing compensation module realizes graph convolutional propagation by constructing a modality adjacency graph, the features of each modality node are mapped to a unified multidimensional space, and a two-layer graph neural network is used for prediction, with the loss function being the mean squared error; The adjacency matrix of the graph neural network is initialized using Pearson correlation coefficient or cosine similarity, or obtained through end-to-end learning during training. The image modal features, electrical modal features, and process modal features are input into the modal loss compensation module, and the reconstructed modal features are output.

[0010] Furthermore, the multimodal cross-attention module is used to interact and fuse image modal features, electrical modal features, process modal features, and reconstruction modal features to output a fused feature vector, including: Image modal features, electrical modal features, process modal features, and remodeling modal features are input into the multimodal cross-attention module, specifically: Each modal feature is concatenated with the modal embedding, and all the concatenation results are combined into an input sequence. The input sequence is input to a multimodal cross-attention module, which outputs a fused feature vector to achieve cross-modal coupling. The multimodal cross-attention module adopts a Transformer Encoder structure and a multi-head attention mechanism.

[0011] Furthermore, The step of inputting the fused feature vector to the multi-scale output module to output multi-scale fused features includes: The fused feature vector is input to the multi-scale output module, which outputs point-level, tile-level, and wafer-level feature representation vectors respectively to obtain multi-scale fused features. The multi-scale output module includes three branches: point-level output, tile-level output, and wafer-level output. The point-level output adopts the U-NetDecoder structure, which includes an encoder-decoder symmetrical structure and skip connections, for point-like physical modeling and atomic-level defect localization; The tile-level output generates multi-dimensional vectors through a three-layer fully connected network for local material analysis and crystal defect statistics. The wafer-level output generates a multi-dimensional vector through a three-layer fully connected network, which is used for statistical modeling, graph management, and wafer yield prediction.

[0012] Furthermore, the method of employing a progressive three-stage training approach to jointly optimize multi-scale fusion features for multi-modal data, in order to achieve cross-modal coupling of semiconductor multi-modal data, includes: A progressive three-stage training approach is adopted to jointly optimize multi-modal data using multi-scale fusion features. The three training stages are the first, second, and third stages, and the specific steps are as follows: In the first stage, the encoder is frozen and pre-trained using a single model to optimize the single-modality feature extractor; In the second stage, the dual-modality is unfrozen, and image + electrical fusion and image + process fusion are used to optimize and form a dual-modality network with image as the main modality and electrical and process as auxiliary modalities. In the third stage, a full unfreezing is performed to achieve joint optimization of multimodal data.

[0013] Furthermore, during training, a joint loss function is used, which is obtained by combining the reconstruction loss, contrast loss, and target task loss. for:

[0014] In the formula, Indicates the losses incurred during reconstruction; Indicates comparative loss; Indicates the loss of the target task; The contrast loss is expressed in NT-Xent form.

[0015] Furthermore, the modal compensation module is extended to a Transformer structure; The graph neural network structure of the modality compensation module is GCNayer (256→128)→ReLU→ConditionalGAN generator; the reconstruction loss is a weighted sum of mean squared error and adversarial loss.

[0016] A cross-modal coupling system for semiconductor multimodal data includes: The data acquisition module is used to acquire multimodal data of semiconductor samples and preprocess each modal data separately; the multimodal data includes image modal data, electrical modal data and process modal data; The feature extraction module is used to extract features from each modal data using a pre-built multimodal sensing encoder network to obtain image modal features, electrical modal features, and process modal features. A feature reconstruction module is used to input image modal features, electrical modal features, and process modal features into a pre-constructed modal loss compensation module to predict and obtain reconstructed modal features; the modal compensation module is constructed based on a graph neural network. The feature fusion module is used to perform information interaction and fusion of image modal features, electrical modal features, process modal features, and reconstruction modal features using a multimodal cross-attention module, in order to output a fused feature vector; The feature output module is used to input the fused feature vector to the multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, tile-level output and wafer-level output; The feature optimization module is used to perform multimodal data joint optimization on multi-scale fused features using a progressive three-stage training method, so as to realize cross-modal coupling of semiconductor multimodal data.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a cross-modal coupling method for semiconductor multimodal data. First, a multimodal perceptual encoder extracts features from image, electrical, and process modes respectively. Then, a modality missing compensation module constructed using a graph neural network reconstructs missing features based on heterogeneous data correlations. Next, a multimodal cross-attention mechanism enables dynamic interaction and fusion between features. The fused features are then input into a multi-scale output module to generate point-level, patch-level, and wafer-level features. Finally, a progressive three-stage training method optimizes cross-modal coupling. The cross-attention module deeply mines the contextual dependencies between modes, solving the problem of shallow fusion in existing methods. The graph neural network compensation module simulates the topological relationships between data for inference-based reconstruction, enhancing the robustness of missing data. The multi-scale output adapts to the dimensional span of semiconductors, while the progressive training optimizes feature extraction, compensation, and fusion in stages, improving scenario adaptability. This method can comprehensively model material and device characteristics, deeply capture potential cross-modal correlations, effectively compensate for data missingness, and achieve structural adaptation for semiconductor-specific tasks, significantly improving fusion expressiveness and robustness.

[0018] Preferably, in this invention, standardized preprocessing steps, including image size unification and filtering, and outlier removal of electrical and process data, ensure consistent quality of multimodal data, provide reliable input for subsequent feature extraction, solve the problems of large dimensional span and inconsistent sampling frequencies of semiconductor data, and enhance the overall stability of the system.

[0019] Preferably, in this invention, modality-specific encoders such as the Swin Transformer are used to process image data, and multilayer perceptrons are used to process electrical and process data. This accurately extracts deep features of each modality, effectively captures spatial structure and temporal patterns, lays an accurate foundation for cross-modal fusion, and improves the pertinence and reliability of feature representation.

[0020] Preferably, in this invention, the compensation module based on graph neural networks uses intermodal topological relationships to infer and reconstruct missing features. It achieves knowledge-driven reconstruction by initializing the adjacency matrix through end-to-end learning or similarity, thereby enhancing the robustness of the algorithm when data is incomplete and ensuring that the fusion performance does not decrease due to the absence of some modalities.

[0021] Preferably, in this invention, the multi-head attention mechanism of the Transformer encoder is used to dynamically model the contextual dependencies between modalities, realize deep information interaction and semantic alignment, solve the problem that existing fusion methods fail to capture potential cross-modal associations, and improve the expressiveness and consistency of the fusion effect.

[0022] Preferably, in this invention, the multi-scale output module generates point-level, block-level, and wafer-level features to adapt to the multi-level tasks of semiconductor inspection, from atomic defect localization to wafer yield prediction. Through the design of a structured decoder, the structural adaptability and task specificity of the model in specific semiconductor scenarios are enhanced.

[0023] Preferably, in this invention, the progressive three-stage training strategy optimizes feature extraction, bimodal fusion, and full-modal collaboration in stages, ensuring that the model gradually learns complex modal relationships, avoiding overfitting, improving fusion depth and scene adaptability, while optimizing training efficiency and model generalization.

[0024] Preferably, in this invention, the joint loss function integrates reconstruction error, contrast loss, and target task loss, promoting cross-modal feature alignment and semantic consistency, enhancing model generalization ability and performance stability, and ensuring that the optimization process fully covers data reconstruction, modal contrast, and the final task objective.

[0025] Preferably, in this invention, the extended compensation module is a Transformer or a conditional generative adversarial network structure to improve the realism and accuracy of missing feature reconstruction. Through the adversarial loss weighting mechanism, it further enhances the robustness and expression depth when data is missing and optimizes the overall fusion quality. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a specific implementation of a cross-modal coupling method for semiconductor multimodal data provided in an embodiment of the present invention. Figure 2 The following is a flowchart of the Transformer structure and multimodal data fusion provided in the embodiments of the present invention; wherein, (a) is a schematic diagram of the Transformer structure; and (b) is a flowchart of the multimodal data fusion process. Figure 3 A flowchart illustrating the three-stage progressive training process provided in this embodiment of the invention; Figure 4 A flowchart illustrating a cross-modal coupling method for semiconductor multimodal data provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a cross-modal coupling system for semiconductor multimodal data provided in an embodiment of the present invention. Detailed Implementation

[0027] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0028] To address the problems mentioned in the background section, this embodiment provides a cross-modal coupling method for semiconductor multimodal data. Specifically, it provides a cross-modal adaptive fusion and characterization extraction method capable of fusing data from multiple sources (such as images, electrical data, and process data). This method is primarily applicable to detection, modeling, and data analysis tasks for various semiconductor crystal materials. The specific objectives are as follows: To achieve deep coupling and fusion of multimodal data and semantic collaborative expression, a modality-aware encoder network is constructed. Combining modality identification and semantic embedding, a cross-modal attention mechanism is introduced to model and share semantic relationships between modalities from different physical sources, enhancing the fusion expression capability. Through a modality importance assessment mechanism, dynamic weights are assigned to each modality during the fusion process, avoiding information redundancy from interfering with key features and improving the system's generalization and interpretability across different tasks. A modality prediction and reconstruction module is introduced to automatically generate latent representations of corresponding modalities in scenarios with missing modalities, ensuring stable operation of the model under various sampling configurations or incomplete data. Through a multi-input, multi-granularity output structure, it meets the multi-scale analysis needs from lattice level, nanoscale to device level, is compatible with multi-source heterogeneous data structures, and is applicable to different data forms such as images, electrical data, and structural parameters.

[0029] The cross-modal coupling method for semiconductor multimodal data provided in this embodiment will be explained in detail below with reference to the accompanying drawings: like Figure 4As shown, this embodiment provides a cross-modal coupling method for semiconductor multimodal data, including: Multimodal data of semiconductor samples are acquired and each modal data is preprocessed separately; the multimodal data includes image modal data, electrical modal data and process modal data; A pre-built multimodal sensing encoder network is used to extract features from each modal data to obtain image modal features, electrical modal features, and process modal features; Image modal features, electrical modal features, and process modal features are input into a pre-constructed modal loss compensation module to predict and obtain reconstructed modal features; the modal compensation module is constructed based on a graph neural network. A multimodal cross-attention module is used to interact and fuse image modal features, electrical modal features, process modal features, and remodeling modal features to output a fused feature vector; The fused feature vector is input to the multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, tile-level output, and wafer-level output; A progressive three-stage training approach is adopted to jointly optimize multi-scale fusion features for multi-modal data, so as to achieve cross-modal coupling of semiconductor multi-modal data.

[0030] like Figure 1 As shown below, the cross-modal coupling method for semiconductor multimodal data provided in this embodiment is used to construct a multimodal feature database for semiconductor materials. Each record in the database contains a fused unified representation vector and an index of the original modal data, which can be used for subsequent retrieval, clustering, modeling, or simulation. Specifically, it includes: Step S1: Multimodal data acquisition and preprocessing; Step S2: Construction of the modality-aware feature extraction module; Step S3: Cross-modal coupling fusion; Step S4: Design of modal missing compensation mechanism; Step S5: Multi-scale fusion feature output; Step S6: Supervised training and optimization.

[0031] The above steps are implemented in the following specific steps. This method is applicable to the intelligent fusion and modeling of multimodal characterization data for third-generation semiconductor materials, such as gallium nitride (GaN), silicon carbide (SiC), and gallium oxide (Ga2O3). The steps are as follows: (1) Multimodal data acquisition and preprocessing: Image mode (such as AFM, SEM), electrical mode (such as resistivity, mobility) and process mode (such as temperature, air pressure) data are acquired from semiconductor samples and normalized respectively; the image modes are scaled to a uniform size.

[0032] (2) Construct a modality-aware encoder network and construct a corresponding feature extraction sub-network for each modality: the image modality uses a WinTransformer encoder, and the electrical and process modality uses a multilayer perceptron (MLP); each encoder outputs modality features and embedded identifier vectors; (3) Construct a cross-modal coupling module, and feed the above-mentioned modal embeddings into the multi-head cross-modality attention mechanism to realize information interaction and fusion between multi-modal features and output a unified fused feature vector; (4) Modal missing compensation mechanism design: a modal occlusion strategy is introduced during the training phase, and a modal dependency graph is constructed through graph neural network (GCN) to embed and reconstruct the occluded modalities; the GCN propagation mechanism is used to transfer feature information between existing modalities to improve the robustness of the model. (5) Multi-scale fusion feature output: The fused feature vector is output as point-level, patch-level and wafer-level feature representation vectors through a multi-task projection head, which can be called by subsequent downstream tasks or displayed in a structured manner.

[0033] (6) Supervised training and optimization: Through progressive three-stage training, by freezing relevant layers, the performance of the network after adding different modalities is optimized one by one. The optimized loss function is obtained by adjusting the coefficients of joint reconstruction loss, contrast loss and target task loss.

[0034] It should be noted that the order of steps S3 and S4 in this embodiment can be interchanged, that is, there are two methods: compensation before fusion or fusion before compensation and then fusion.

[0035] The above methods will be explained in more detail below: For example, step 1 includes: (1a) Collect multimodal data from third-generation semiconductor samples (such as epitaxial GaN materials): image modal data M1: AFM, SEM grayscale images; electrical modal data M2: surface potential, mobility, resistivity; process modal data M3: temperature, pressure, reaction time, gas ratio structured process parameters; (1b) The image modal data M1 is scaled to 224x224 by bilinear interpolation and normalized after Gaussian filtering; (1c) Electrical and process parameters are standardized using Z-score and passed through... The criteria for removing outliers are as follows: electrical modes are represented by 1×10-dimensional vectors, and process modes are represented by 1×8-dimensional vectors.

[0036] For example, step 2 includes: (2a) Image modality feature extractor: The SwinTransformerTiny model is used as the backbone network, with the following configuration parameters: patch_size=4, embed_dim=96, depths=[2,2,6,2], heads=[3,6,12,24]; and a convolutional block attention module (CBAM) is connected to the output of the Transformer to output a 768-dimensional feature vector F1. (2b) Electrical process feature extractor: Electrical modes are extracted using the FC(10→32)→ReLU→FC(32→64)→output F2∈R^{64} network; process modes are extracted using the FC(8→16)→ReLU→FC(16→32)→FC(32→64)→output F3∈R^{64} network. (2c) Feature Alignment and Embedding: Image feature projection is reduced from 768 dimensions to 256 dimensions, while electrical and process features are increased from 64 dimensions to 256 dimensions, resulting in a unified 256-dimensional feature vector. A 16-dimensional modality identifier vector Mi is generated through a learnable embedding layer, with each modality corresponding to a unique identifier vector. In this embodiment, the SwinTransformer encoder is configured as follows: patch_size=4, embed_dim=96, depths=[2,2,6,2], heads=[3,6,12,24], and outputs 768-dimensional image features through a convolutional block attention module (CBAM).

[0037] like Figure 2 As shown, specifically as follows Figure 2 As shown in Figures (a) and (b), step 3, by way of example, includes: (3a) Concatenate the modal features and modal embeddings: Combine all the splicing results into the input sequence. .

[0038] (3b) Implement cross-modal coupling using TransformerEncoder, with the following configuration parameters: num_layers=2, num_heads=4, hidden_dim=512, dropout=0.1, and output... .

[0039] In this embodiment, the multimodal cross-attention module is a TransformerEncoder structure, which adopts a multi-head attention mechanism and includes two encoding layers. Each layer has eight attention heads and a 512-dimensional hidden layer, and the output fusion vector has a dimension of 512.

[0040] For example, step 4 includes: (4a) A graph neural network (GCN) is used to construct a graph of intermodal correlations, and existing modal features are used to predict the feature representation of missing modes to construct a dynamic adjacency matrix. , where θ is a learnable parameter, d=256; (4b) A typical graph convolutional reconstruction network is: GCNLayer (256→128) → ReLU → ConditionalGAN (generator) → Output reconstructed features F̂; where electrical data is reconstructed based on the most relevant process mode. In this embodiment, the modality missing compensation module achieves graph convolution propagation by constructing a modality adjacency graph. The features of each modality node are mapped to a unified 128-dimensional space, and a two-layer GCN network is used for prediction. The loss function is the mean squared error.

[0041] The modality missing compensation module can be extended to a Transformer structure to realize modality reconstruction based on the attention mechanism, where the query vector of the missing modality is aggregated from the existing modalities.

[0042] The adjacency matrix of GCN is initialized using Pearson correlation coefficient or cosine similarity, or it can be learned end-to-end during training.

[0043] The graph convolutional network structure is GCNayer (256→128) → ReLU → ConditionalGAN generator; when reconstructing electrical data with process modes as conditions, the generator input is a concatenated vector of process features and random noise. The reconstruction loss is a weighted sum of mean squared error (MSE) and adversarial loss.

[0044] For example, step 5 includes: (5a) will The input multi-scale output module can be divided into point-level output, tile-level output, and wafer-level output according to the output task. (5b) The typical network structure for point-level output is U-NetDecoder(Z fused ), output dimension is It is mainly used for point-based physical modeling.

[0045] (5c) The typical network structure for tile-level output is FC (512→256→128), with an output dimension of It is mainly used for local material analysis.

[0046] (5d) The typical network structure for wafer-level output is FC (512→256→64), with an output dimension of It is mainly used for statistical modeling and graph management.

[0047] In this embodiment, the multi-scale output module includes three branches: the point output branch includes two fully connected networks (512→128→32), the block output is (512→256→128), and the wafer output is (512→256→64).

[0048] The point-level output adopts the U-NetDecoder structure, which includes an encoder-decoder symmetric structure and skip connections, and outputs a 32-dimensional vector for atomic-level defect localization; the tile-level output generates a 128-dimensional vector through a three-layer fully connected network (512→256→128) for local crystal defect statistics; the wafer-level output generates a 64-dimensional vector through a three-layer fully connected network (512→256→64) for wafer yield prediction.

[0049] For example, step 6 includes: (6a) such as Figure 3 Through progressive three-stage training, the encoder is frozen in the first stage, and a single model is used for pre-training. In the second stage, the dual-modality network is unfrozen, and image + electrical fusion and image + process fusion are used to optimize the dual-modality network with image as the main modality and electrical and process as auxiliary modalities, respectively. In the third stage, the entire network is unfrozen, and multimodal data is jointly optimized.

[0050] (6b) Loss function design: The optimized loss function is obtained by adjusting the coefficients of the joint reconstruction loss, contrast loss, and target task loss. Because there is a dimensional inconsistency in the data between different modes of semiconductor detection, in order to align features of different dimensions in a unified space, the contrast loss can be implemented using the NT-Xent form: In the formula, Indicates the similarity between positive sample pairs; This represents the temperature parameter.

[0051] like Figure 1 The cross-modal coupling method for semiconductor multimodal data provided in the above embodiments is applied and implemented as follows: This embodiment uses gallium nitride (GaN) epitaxial thin film samples as the research object to illustrate the application process and effect of the cross-modal coupling method of semiconductor multimodal data of the present invention in the analysis of actual semiconductor materials.

[0052] Step S1: Multimodal data acquisition and preprocessing: In the experiment, multiple GaN thin film samples grown using metal-organic chemical vapor deposition (MOCVD) were selected. Multiple wafers were prepared under different process conditions (temperature, V / III ratio, reaction time) to verify the applicability and effectiveness of the method of this invention in processing complex multimodal data. Each sample underwent various characterization methods, including but not limited to surface structure images, electrical performance tests, and process records. Data preprocessing was performed using Python+NumPy+OpenCV+SciPy. Atomic force microscopy (AFM) and scanning electron microscopy (SEM) were used to observe the surface morphology of the samples. The obtained grayscale images were uniformly adjusted to 224×224 pixels and Z-score normalized. Image enhancement was performed using the Albumentations library to ensure the model had sufficient generalization ability. Electrical parameters such as electron mobility (μ), carrier concentration (n), and resistivity (ρ) of each sample were obtained using a Hall effect testing instrument. All parameters were normalized to the [0,1] range and used as structured floating-point feature inputs. The process modes are recorded in real time during the preparation process, including reaction temperature, reaction pressure, gas flow rate (NH3, TMGa) and reaction time, etc. A total of 8-dimensional structured process vectors are extracted and standardized.

[0053] Step S2: Construction of the modality-aware feature extraction module: Using the PyTorch platform, three modality-aware sub-networks were designed: For image modalities (such as AFM or SEM images), the SwinTransformer from the Visual Transformer architecture was used as the encoder, which features multi-scale modeling and window attention mechanisms, effectively capturing local and global features of crystal surface morphology. The input to this network is a 224×224 pixel grayscale image, and the output is a high-dimensional embedding vector representing the semantic information of the image.

[0054] For electrical modes (such as resistivity and mobility) and process modes (such as temperature and pressure), since they are structured numerical data, a two-layer multilayer perceptron (MLP) is used for modeling. The inherent distribution and numerical relationship are extracted through nonlinear transformation and hidden layer mapping, and a fixed-length vector representation is output.

[0055] Each modality's output feature is accompanied by a "modality embedding" vector, which serves as a modality identifier and is used in subsequent fusion to guide the model in identifying the physical origin and characteristics of each modality.

[0056] Step S3: Cross-modal coupling fusion: The data representations from different modalities are effectively fused to extract collaborative information between modalities. The feature vectors output by each modal encoder in step (2) are concatenated with the modal embedding vectors to form a complete set of modal feature sequences.

[0057] This sequence is input into a Transformer-based multi-head attention module, namely a cross-modal Transformer encoder. This module uses a multi-head attention mechanism to simulate the information transfer process between modalities and automatically learn the correlation and complementarity between different modalities.

[0058] This module constructs a multi-layered information interaction channel through two stacked attention encoding layers, fusing modal feature sequences into a unified high-dimensional representation vector. Features output from all modalities are concatenated with their modal embedding vectors and fed into the Cross-ModalityTransformer, a two-layer encoder containing eight attention heads and a 512-dimensional hidden layer. This fused representation vector possesses structural, compositional, electrical, and technological semantic information shared by all modalities, serving as the core feature input for subsequent tasks.

[0059] Step S4: Design of Modal Missing Compensation Mechanism: A Graph Neural Network (GCN) is introduced as a modality compensation module. A modality dependency graph is constructed based on the co-occurrence frequency and physical coupling relationship between samples. A two-layer graph convolutional network is used to propagate existing modality information and predict the feature embeddings of missing modalities. During the training phase, a random masking strategy is used to simulate modality missing scenarios, improving the model's robustness to data gaps. The prediction error of the compensation module is included in the total training loss as a metric of mean squared error (MSE).

[0060] Step S5: Multi-scale fusion feature output: The output fused feature vectors are fed into three output branches: point-level (32-dimensional), tile-level (128-dimensional), and wafer-level (64-dimensional), for subsequent tasks such as modeling, material clustering, and yield analysis. To improve training stability and accuracy, the AdamW model optimizer is used, with an initial learning rate set to 1×10⁻⁶. -4 The model is dynamically adjusted using a CosineAnnealing strategy. The loss function consists of four joint losses: modality consistency loss, modality correlation enhancement loss, modality reconstruction loss, and target task loss, which jointly optimize the model's fusion capability and missing data recovery performance.

[0061] Step S6: Supervised Training and Optimization The training process employs a progressive three-stage approach. In the first stage, the encoder is frozen, and a single model is used for pre-training. In the second stage, the dual-modality network is unfrozen, and image-plus-electricity fusion and image-plus-process fusion are used to optimize the dual-modality network, with image as the primary modality and electricity and process as secondary modalities, respectively. In the third stage, the network is fully unfrozen, and multi-modal data is jointly optimized. The optimized loss function is obtained by adjusting the coefficients of the joint reconstruction loss, contrast loss, and target task loss. .

[0062] In this embodiment, the training phase employs a progressive three-stage training approach. Stage one freezes the Encoder and optimizes only the single-modal feature extractor. Stage two focuses on the image modality, optimizing the "image + electrical" and "image + process" dual-modal fusion networks respectively. Stage three unfreezes all parameters and performs full-modal optimization.

[0063] For example, the method provided in this embodiment can be integrated into a smart characterization platform for semiconductor materials, providing functions for the acquisition, fusion, storage, visualization, and structured export of multimodal raw data. The method further includes a modality adaptive importance ranking step, which dynamically evaluates the relative contribution of different modalities in the current task based on the cross-attention weights during training, in order to support modality selection and modality compression.

[0064] like Figure 5 As shown, this embodiment also provides a cross-modal coupling system for semiconductor multimodal data, including: a data acquisition module for acquiring multimodal data of a semiconductor sample and preprocessing each modal data separately; the multimodal data includes image modal data, electrical modal data, and process modal data; a feature extraction module for extracting features from each modal data using a pre-built multimodal perception encoder network to obtain image modal features, electrical modal features, and process modal features; and a feature reconstruction module for inputting the image modal features, electrical modal features, and process modal features into a pre-built modal loss compensation module to predict the reconstructed modal. Features; the modal compensation module is constructed based on a graph neural network; the feature fusion module is used to use a multimodal cross-attention module to interact and fuse image modal features, electrical modal features, process modal features, and remodeling modal features to output a fused feature vector; the feature output module is used to input the fused feature vector to a multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, patch-level output, and wafer-level output; the feature optimization module is used to use a progressive three-stage training method to jointly optimize the multi-scale fused features for multimodal data to achieve cross-modal coupling of semiconductor multimodal data.

[0065] The present invention also provides a cross-modal coupling device for semiconductor multimodal data, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the cross-modal coupling method for semiconductor multimodal data.

[0066] When the processor executes the computer program, it implements the aforementioned steps for cross-modal coupling of semiconductor multimodal data, for example: acquiring multimodal data of a semiconductor sample and preprocessing each modal data separately; the multimodal data includes image modal data, electrical modal data, and process modal data; using a pre-built multimodal sensing encoder network to extract features from each modal data to obtain image modal features, electrical modal features, and process modal features; inputting the image modal features, electrical modal features, and process modal features into a pre-built modality loss compensation module to predict... The modal features are remodeled; the modal compensation module is constructed based on a graph neural network; a multimodal cross-attention module is used to interact and fuse image modal features, electrical modal features, process modal features, and remodeled modal features to output a fused feature vector; the fused feature vector is input to a multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, patch-level output, and wafer-level output; a progressive three-stage training method is used to jointly optimize the multi-scale fused features for multimodal data to achieve cross-modal coupling of semiconductor multimodal data.

[0067] Exemplarily, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing preset functions, the instruction segments describing the execution process of the computer program in the cross-modal coupling device for semiconductor multimodal data. For example, the computer program can be divided into a data acquisition module, a feature extraction module, a feature reconstruction module, a feature fusion module, a feature output module, and a feature optimization module; the specific functions of each module are as follows: the data acquisition module is used to acquire multimodal data of semiconductor samples and preprocess each modal data separately; the multimodal data includes image modal data, electrical modal data, and process modal data; the feature extraction module is used to extract features from each modal data using a pre-built multimodal sensing encoder network to obtain image modal features, electrical modal features, and process modal features; the feature reconstruction module is used to input the image modal features, electrical modal features, and process modal features into the pre-built... A modality missing compensation module is used to predict and obtain remodeled modal features; the modality compensation module is constructed based on a graph neural network; a feature fusion module is used to use a multimodal cross-attention module to interact and fuse image modal features, electrical modal features, process modal features, and remodeled modal features to output a fused feature vector; a feature output module is used to input the fused feature vector to a multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, patch-level output, and wafer-level output; a feature optimization module is used to use a progressive three-stage training method to jointly optimize the multi-scale fused features for multimodal data to achieve cross-modal coupling of semiconductor multimodal data.

[0068] The cross-modal coupling device for semiconductor multimodal data can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The cross-modal coupling device for semiconductor multimodal data may include, but is not limited to, processors and memory. Those skilled in the art will understand that the above are examples of cross-modal coupling devices for semiconductor multimodal data and do not constitute a limitation on such devices. The device may include more components than described above, or combine certain components, or different components. For example, the cross-modal coupling device for semiconductor multimodal data may also include input / output devices, network access devices, buses, etc.

[0069] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. The processor is the control center for the cross-modal coupling of the semiconductor multimodal data, connecting various parts of the entire semiconductor multimodal data cross-modal coupling device through various interfaces and lines.

[0070] The memory can be used to store the computer program and / or modules. The processor implements various functions of the cross-modal coupling device for semiconductor multimodal data by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory.

[0071] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.). The data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0072] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the cross-modal coupling method for semiconductor multimodal data.

[0073] If the modules / units of the cross-modal coupling system for semiconductor multimodal data are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0074] Based on this understanding, the present invention can implement all or part of the processes in the above-described cross-modal coupling method for semiconductor multimodal data, which can also be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described cross-modal coupling method for semiconductor multimodal data. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or a preset intermediate form, etc.

[0075] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0076] It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0077] The above embodiments are merely one of the implementation methods for achieving the technical solution of the present invention. The scope of protection claimed by the present invention is not limited to this embodiment, but also includes any variations, substitutions and other implementation methods that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for cross-modal coupling of semiconductor multimodal data, characterized in that, include: Multimodal data of semiconductor samples are acquired and each modal data is preprocessed separately; the multimodal data includes image modal data, electrical modal data and process modal data; A pre-built multimodal sensing encoder network is used to extract features from each modal data to obtain image modal features, electrical modal features, and process modal features; Image modal features, electrical modal features, and process modal features are input into a pre-constructed modal loss compensation module to predict and obtain reconstructed modal features; the modal compensation module is constructed based on a graph neural network. A multimodal cross-attention module is used to interact and fuse image modal features, electrical modal features, process modal features, and remodeling modal features to output a fused feature vector; The fused feature vector is input to the multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, tile-level output, and wafer-level output; A progressive three-stage training approach is adopted to jointly optimize multi-scale fusion features for multi-modal data, so as to achieve cross-modal coupling of semiconductor multi-modal data.

2. The cross-modal coupling method for semiconductor multimodal data according to claim 1, characterized in that, The process of acquiring multimodal data from semiconductor samples and preprocessing each modal data separately includes: Image modal data, electrical modal data, and process modal data were separately acquired from the semiconductor sample. The image modal data included AFM and SEM grayscale images; the electrical modal data included surface potential, mobility, and resistivity; and the process modal data included temperature, pressure, reaction time, and gas ratio structured process parameters. The image modal data is scaled to a uniform preset size using bilinear interpolation and then normalized after Gaussian filtering. Electrical modal data and process modal data were standardized using Z-score, and then... Criteria for removing outliers.

3. The cross-modal coupling method for semiconductor multimodal data according to claim 1, characterized in that, The pre-constructed multimodal sensing encoder network extracts features from each modal data to obtain image modal features, electrical modal features, and process modal features, including: For each modal data, a corresponding feature extraction sub-network is constructed to obtain the multimodal sensing encoder network; among them, the image modal data adopts the Swing Transformer encoder; the electrical modal data and the process modal data both adopt the multilayer perceptron. Image modal data is input into the Swing Transformer encoder, and image modal features are output through the convolutional block attention module; Electrical modal data and process modal data are input into a multilayer perceptron, which outputs electrical modal features and process modal features.

4. The cross-modal coupling method for semiconductor multimodal data according to claim 1, characterized in that, The step of inputting image modal features, electrical modal features, and process modal features into a pre-constructed modal loss compensation module to predict and obtain reconstructed modal features includes: A modality missing compensation module is constructed; wherein, the modality missing compensation module realizes graph convolutional propagation by constructing a modality adjacency graph, the features of each modality node are mapped to a unified multidimensional space, and a two-layer graph neural network is used for prediction, with the loss function being the mean squared error; The adjacency matrix of the graph neural network is initialized using Pearson correlation coefficient or cosine similarity, or obtained through end-to-end learning during training. The image modal features, electrical modal features, and process modal features are input into the modal loss compensation module, and the reconstructed modal features are output.

5. The cross-modal coupling method for semiconductor multimodal data according to claim 1, characterized in that, The multimodal cross-attention module is used to interact and fuse image modal features, electrical modal features, process modal features, and reconstruction modal features to output a fused feature vector, including: Image modal features, electrical modal features, process modal features, and remodeling modal features are input into the multimodal cross-attention module, specifically: Each modal feature is concatenated with the modal embedding, and all the concatenation results are combined into an input sequence. The input sequence is input to a multimodal cross-attention module, which outputs a fused feature vector to achieve cross-modal coupling. The multimodal cross-attention module adopts a Transformer Encoder structure and a multi-head attention mechanism.

6. The cross-modal coupling method for semiconductor multimodal data according to claim 1, characterized in that, The step of inputting the fused feature vector to the multi-scale output module to output multi-scale fused features includes: The fused feature vector is input to the multi-scale output module, which outputs point-level, tile-level, and wafer-level feature representation vectors respectively to obtain multi-scale fused features. The multi-scale output module includes three branches: point-level output, tile-level output, and wafer-level output. The point-level output adopts the U-NetDecoder structure, which includes an encoder-decoder symmetrical structure and skip connections, for point-like physical modeling and atomic-level defect localization; The tile-level output generates multi-dimensional vectors through a three-layer fully connected network for local material analysis and crystal defect statistics. The wafer-level output generates a multi-dimensional vector through a three-layer fully connected network, which is used for statistical modeling, graph management, and wafer yield prediction.

7. The cross-modal coupling method for semiconductor multimodal data according to claim 1, characterized in that, The method employs a progressive three-stage training approach to jointly optimize multi-scale fusion features for multi-modal data, thereby achieving cross-modal coupling of semiconductor multi-modal data. This includes: A progressive three-stage training approach is adopted to jointly optimize multi-modal data using multi-scale fusion features. The three training stages are the first, second, and third stages, and the specific steps are as follows: In the first stage, the encoder is frozen and pre-trained using a single model to optimize the single-modality feature extractor; In the second stage, the dual-modality is unfrozen, and image + electrical fusion and image + process fusion are used to optimize and form a dual-modality network with image as the main modality and electrical and process as auxiliary modalities. In the third stage, a full unfreezing is performed to achieve joint optimization of multimodal data.

8. The cross-modal coupling method for semiconductor multimodal data according to claim 7, characterized in that, During training, a joint loss function is used, which is obtained by combining reconstruction loss, contrast loss, and target task loss. for: In the formula, Indicates the losses incurred during reconstruction; Indicates comparative loss; Indicates the loss of the target task; The contrast loss is expressed in NT-Xent form.

9. The cross-modal coupling method for semiconductor multimodal data according to claim 1, characterized in that, The modal compensation module is extended to a Transformer structure; The graph neural network structure of the modality compensation module is GCNayer (256→128)→ReLU→ConditionalGAN generator; the reconstruction loss is a weighted sum of mean squared error and adversarial loss.

10. A cross-modal coupling system for semiconductor multimodal data, characterized in that, include: The data acquisition module is used to acquire multimodal data of semiconductor samples and preprocess each modal data separately; the multimodal data includes image modal data, electrical modal data and process modal data; The feature extraction module is used to extract features from each modal data using a pre-built multimodal sensing encoder network to obtain image modal features, electrical modal features, and process modal features. A feature reconstruction module is used to input image modal features, electrical modal features, and process modal features into a pre-constructed modal loss compensation module to predict and obtain reconstructed modal features; the modal compensation module is constructed based on a graph neural network. The feature fusion module is used to perform information interaction and fusion of image modal features, electrical modal features, process modal features, and reconstruction modal features using a multimodal cross-attention module, in order to output a fused feature vector; The feature output module is used to input the fused feature vector to the multi-scale output module to output multi-scale fused features; the multi-scale fused features include point-level output, tile-level output and wafer-level output; The feature optimization module is used to perform multimodal data joint optimization on multi-scale fused features using a progressive three-stage training method, so as to realize cross-modal coupling of semiconductor multimodal data.

Images