A multimodal relationship discovery method in the semiconductor field

Abstract

The present invention relates to a multimodal relationship discovery method in the semiconductor field, comprising: acquiring image data and text data; generating a scene graph according to the image data; generating a dependency syntax tree according to the text data; encoding the text data using an encoder to obtain text encoding vectors corresponding to entities in the text data; generating corresponding visual semantic vectors according to entity nodes in the scene graph; fusing the scene graph with the dependency syntax tree according to the similarity between the entities to obtain a first association graph; splicing the visual semantic vector and the text encoding vector according to the similarity to obtain a first vector; splicing the first vector with a second vector corresponding to an entity in the first association graph according to the similarity to obtain a third vector; acquiring a fourth vector corresponding to a known entity, splicing the third vector and the fourth vector and inputting the spliced ​​vectors into a relationship prediction model to obtain a relationship between the third vector and the entities corresponding to the fourth vector.

Description

Technical Field

[0001] The present invention relates to the field of semiconductor manufacturing, and in particular to a multimodal relationship discovery method in the semiconductor field. Background Art

[0002] During the semiconductor manufacturing process, engineers create semi-structured text based on wafer inspection results to record defects and failures, along with their causes. Simultaneously, defect inspections and electrical tests related to wafer defect detection during production also record wafer inspection results in the form of images.

[0003] For the text and image data in the above-mentioned semiconductor field, existing technologies often only analyze and mine a single type of data, while ignoring the possible relationships between different types of data. This makes it difficult to discover the relationships between entities contained in multimodal data. Summary of the Invention

[0004] One or more embodiments of this specification describe a multimodal relationship discovery method in the semiconductor field, which simultaneously models and analyzes image and text data. While obtaining the relationship between entities in the same type of data, it can also discover the relationship between entities contained in different types of data to obtain better relationship discovery results.

[0005] This specification provides a multimodal relationship discovery method in the semiconductor field, including:

[0006] Acquiring image data and text data, wherein the image data and text data are data related to defect detection in a semiconductor manufacturing process;

[0007] generating a scene graph based on the image data, wherein the scene graph includes entities in the image and relationships between the entities;

[0008] Generating a dependency syntax tree according to the text data, wherein the dependency syntax tree includes entities in the text and relationships between entities;

[0009] Encode the text data using an encoder to obtain a text encoding vector corresponding to an entity in the text data;

[0010] Generate corresponding visual semantic vectors according to the entity nodes in the scene graph;

[0011] Fusing the scene graph and the dependency syntax tree according to the similarity between entities to obtain a first association graph;

[0012] Concatenating the visual semantic vector and the text encoding vector according to similarity to obtain a first vector;

[0013] Concatenate the first vector and the second vector corresponding to the entity in the first association graph according to similarity to obtain a third vector;

[0014] A fourth vector corresponding to a known entity is obtained, and the third vector and the fourth vector are concatenated and input into a relationship prediction model to obtain a relationship between the entities corresponding to the third vector and the fourth vector.

[0015] In one possible implementation, the image data includes multiple of the following: a wafer image with defect category labels, a wafer defect detection image, a photo of a wafer defect, and an electrical test result image;

[0016] The text data includes multiple of the following: semiconductor equipment user manuals, semiconductor equipment troubleshooting manuals, and fault analysis documents written by semiconductor engineers;

[0017] The entities in the image and / or text include multiple of the following: equipment for producing wafers, wafers, sensors, die failure categories, wafer defect categories, wafer defect coordinates, and production failure categories.

[0018] In a possible implementation, the scene graph and the dependency syntax tree are fused according to the similarity between entities to obtain a first association graph, including:

[0019] Calculating a first similarity between a first entity in the scene graph and a second entity in the dependency syntax tree based on a first matrix representation corresponding to the scene graph and a second matrix representation corresponding to the dependency syntax tree;

[0020] When the first similarity is greater than a preset first threshold, the first entity and the second entity are fused to obtain a first association graph.

[0021] In a possible implementation, the visual semantic vector and the text encoding vector are concatenated according to similarity to obtain a first vector, including:

[0022] Calculating a second similarity between the visual semantic vector and the text encoding vector;

[0023] When the second similarity is greater than a preset second threshold, the visual semantic vector and the text encoding vector are concatenated to obtain a first vector.

[0024] In a possible implementation, a method for calculating the second similarity includes:

[0025] Calculating the Euclidean distance between the visual semantic vector and the text encoding vector to obtain a second similarity; or

[0026] Calculating the Manhattan distance between the visual semantic vector and the text encoding vector to obtain a second similarity; or

[0027] Calculating a Chebyshev distance between the visual semantic vector and the text encoding vector to obtain a second similarity; or

[0028] The cosine similarity between the visual semantic vector and the text encoding vector is calculated to obtain a second similarity.

[0029] In a possible implementation, concatenating the first vector with the second vector corresponding to the entity in the first association graph according to similarity to obtain a third vector includes:

[0030] calculating a third similarity between the first vector and the second vector;

[0031] When the third similarity is greater than a preset third threshold, the first vector and the second vector are concatenated to obtain a third vector.

[0032] In a possible implementation, a method for calculating the third similarity includes:

[0033] Calculating the Euclidean distance between the first vector and the second vector to obtain a third similarity; or

[0034] Calculating the Manhattan distance between the first vector and the second vector to obtain a third similarity; or

[0035] calculating a Chebyshev distance between the first vector and the second vector to obtain a third similarity; or

[0036] Calculate the cosine similarity between the first vector and the second vector to obtain a third similarity.

[0037] In one possible implementation, generating a scene graph according to the image data includes:

[0038] The image data is input into a regional convolutional neural network to obtain a scene graph.

[0039] In one possible implementation, the encoder is a BERT encoder or a BiLSTM encoder.

[0040] In a possible implementation, the relationship prediction model is composed of a multi-layer perceptron and a Softmax function.

[0041] The present invention proposes a multimodal relationship discovery method in the semiconductor field, which simultaneously models and analyzes image and text data. While obtaining the relationship between entities in the same type of data, it can also discover the relationship between entities contained in different types of data to obtain better relationship discovery results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 This is a framework diagram of a multimodal relationship discovery method in the semiconductor field disclosed in an embodiment of the present invention;

[0044] Figure 2 This is a flowchart of a multimodal relationship discovery method in the semiconductor field disclosed in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a scene graph disclosed in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a dependency syntax tree disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0048] According to one embodiment, Figure 1 A framework diagram of a multimodal relationship discovery method in the semiconductor field is shown. Figure 1 As shown in FIG, the framework used by the method mainly includes three parts: graph structure representation / text semantic representation, graph structure alignment / semantic alignment, and feature fusion and relationship discovery.

[0049] First, a corresponding scene graph is generated based on the input image data, and a corresponding dependency syntax tree is generated based on the input text data. At the same time, an encoder is used to encode the text data to obtain a text encoding vector. Then, the entities in the generated scene graph and the dependency syntax tree are aligned according to similarity, and the nodes in the scene graph are converted into corresponding visual semantic vectors, and the visual semantic vectors and the text encoding vectors are aligned according to similarity. Finally, the result vectors after the two alignments are aligned again according to similarity to obtain a vector representation of the entity that simultaneously integrates text content and image content. After the vector representation of the entity is spliced ​​with the vector representation of the known entity, it is input into the relationship prediction model to obtain the relationship prediction result between the entity and the known entity.

[0050] The following will further explain and illustrate specific embodiments with reference to the accompanying drawings, which do not constitute a limitation on the embodiments of the present invention.

[0051] Figure 2 This is a flow chart of a method for discovering multimodal relationships in the semiconductor field disclosed in an embodiment of the present invention. Figure 2 As shown, the method at least includes: step 201, obtaining image data and text data, wherein the image data and text data are data related to defect detection in a semiconductor manufacturing process; step 202, generating a scene graph according to the image data, wherein the scene graph includes entities in the image and relationships between entities; step 203, generating a dependency syntax tree according to the text data, wherein the dependency syntax tree includes entities in the text and relationships between entities; step 204, encoding the text data using an encoder to obtain text encoding vectors corresponding to entities in the text data; step 205, generating a text encoding vector according to entities in the scene graph Entity node, generate the corresponding visual semantic vector; step 206, fuse the scene graph and the dependency syntax tree according to the similarity between the entities to obtain a first association graph; step 207, splice the visual semantic vector and the text encoding vector according to the similarity to obtain a first vector; step 208, splice the first vector with the second vector corresponding to the entity in the first association graph according to the similarity to obtain a third vector; step 209, obtain the fourth vector corresponding to the known entity, splice the third vector and the fourth vector and input them into the relationship prediction model to obtain the relationship between the entities corresponding to the third vector and the fourth vector.

[0052] In step 201 , image data and text data are acquired, where the image data and text data are data related to defect detection in a semiconductor manufacturing process.

[0053] The image data includes multiple of the following: wafer images with defect category labels, wafer defect detection images, photos of wafer defects, and electrical test result images.

[0054] A wafer image with defect category labels is a wafer image with defect category labels obtained after defect detection. In some embodiments, due to limited defect detection capabilities, the wafer image with defect category labels obtained through defect detection is a weakly annotated wafer image with incomplete annotations and limited annotation accuracy.

[0055] A wafer defect inspection image shows the results of defect inspection on a wafer image. Defect inspection is a commonly used non-destructive testing method used to detect defects on the chip surface and within the material. These defects can include crystal defects, metal impurities, chemical contaminants, and structural defects. Defect inspection is typically performed using microscopes, SEMs, laser scanning, and other technologies.

[0056] The electrical test results are images of wafers undergoing electrical testing. Electrical testing is a method used to test chip performance, detecting defects by measuring the chip's electrical properties. Electrical testing can detect defects related to electrical performance, such as resistance, capacitance, and leakage. Electrical testing is typically performed using semiconductor test equipment.

[0057] The text data includes multiple of the following: semiconductor equipment user manuals, semiconductor equipment troubleshooting manuals, and failure analysis documents written by semiconductor engineers.

[0058] The content of the failure analysis document written by semiconductor engineers can include one or more of the following: problem description, impact, data description, root cause, defect limitation measures, and next step processing plan.

[0059] In a specific embodiment, a failure analysis document written by a semiconductor engineer may be as follows:

[0060] "Problem description: Wf 728of Lotxxx show massive Bin-number failure.

[0061] Impact: xxx yield loss on impact wafer.

[0062] Data description: (omitted)

[0063] Root cause: scratch during CMP.

[0064] Defect restriction measures: None

[0065] Next step: (omitted)"

[0066] In step 202 , a scene graph is generated based on the image data, wherein the scene graph includes entities in the image and relationships between entities.

[0067] A scene graph is a graphical representation of entities and their relationships in a graphical scene. It is a data structure used to represent visual scenes, and usually contains all entities in the scene and information such as their spatial and semantic relationships. In a scene graph, each entity can be considered a node, and the relationships between entities are connected by edges. In a specific embodiment, a scene graph corresponding to a wafer map can be as follows: Figure 3 shown.

[0068] In one embodiment, scene graph generation (SGG) can use region convolutional neural networks (R-CNN) to generate a corresponding scene graph based on input image data.

[0069] The entities in the image include multiple of the following: equipment for producing wafers, wafers, sensors, die failure categories, wafer defect categories, wafer defect coordinates, and production fault categories.

[0070] In step 203, a dependency syntax tree is generated based on the text data. The dependency syntax tree includes entities in the text and relationships between entities.

[0071] A dependency syntax tree is a structured representation of the dependency relationships between words in a sentence. It represents each word in the sentence as a node and represents the dependency relationships between different words through directed edges. These dependency relationships are usually expressed as grammatical relationships between the central word and its modified words, such as subject-predicate relationships, attributive relationships, object relationships, etc. In one embodiment, the nodes in the dependency syntax tree represent entities in the semiconductor production process. In a specific embodiment, the dependency syntax tree corresponding to a fault analysis document can be as follows: Figure 4 shown.

[0072] In one embodiment, a method for generating a dependency syntax tree based on text data may include the following steps: word segmentation, part-of-speech tagging, syntax analysis, and syntax tree visualization.

[0073] The entities in the text include multiple of the following: equipment for producing wafers, wafers, sensors, die failure categories, wafer defect categories, wafer defect coordinates, and production fault categories.

[0074] In step 204, an encoder is used to encode the text data to obtain a text encoding vector corresponding to an entity in the text data.

[0075] The encoder may be any encoder, for example, a BERT (Bidirectional Encoder Representation from Transformers) encoder or a BiLSTM (Bi-directional Long Short-Term Memory) encoder.

[0076] In step 205, a corresponding visual semantic vector is generated according to the entity nodes in the scene graph.

[0077] When generating the scene graph in step 202, the step of converting the entities in the image into vectors is already included. Here, the vector corresponding to the entity node can be directly extracted, which is called a visual semantic vector.

[0078] In step 206, the scene graph and the dependency syntax tree are fused according to the similarity between entities to obtain a first association graph.

[0079] Specifically, based on the first matrix representation corresponding to the scene graph and the second matrix representation corresponding to the dependency syntax tree, the first similarity between the first entity in the scene graph and the second entity in the dependency syntax tree is calculated; when the first similarity is greater than a preset first threshold, the first entity and the second entity are fused to obtain a first association graph.

[0080] In one embodiment, the first similarity is obtained by calculating the cosine similarity between the first matrix and the second matrix.

[0081] In a specific embodiment, Figure 3 The scene graph shown is similar to Figure 4 If the first similarity between the "Wf728" nodes respectively contained in the dependency syntax tree shown is greater than the first threshold, the nodes corresponding to the two entities are fused.

[0082] In step 207 , the visual semantic vector and the text encoding vector are concatenated according to similarity to obtain a first vector.

[0083] Specifically, a second similarity between the visual semantic vector and the text encoding vector is calculated; when the second similarity is greater than a preset second threshold, the visual semantic vector and the text encoding vector are concatenated to obtain a first vector.

[0084] In one embodiment, the second similarity is obtained by calculating the Euclidean distance, Manhattan distance, or Chebyshev distance between the visual semantic vector and the text encoding vector.

[0085] For vector X=(x1,x2,…,x n ) and vector Y=(y1,y2,…,y n ), the Euclidean distance between vector X and vector Y is calculated as shown in formula (1):

[0086]

[0087] The calculation method of the Manhattan distance between vector X and vector Y is shown in formula (2):

[0088]

[0089] The calculation method of the Chebyshev distance between vector X and vector Y is shown in formula (3):

[0090]

[0091] In another embodiment, the second similarity is obtained by calculating the cosine similarity between the visual semantic vector and the text encoding vector.

[0092] The calculation method of the cosine similarity between vector X and vector Y is shown in formula (4):

[0093]

[0094] In step 208 , the first vector is concatenated with the second vector corresponding to the entity in the first association graph according to similarity to obtain a third vector.

[0095] Specifically, a third similarity between the first vector and the second vector is calculated; when the third similarity is greater than a preset third threshold, the first vector and the second vector are concatenated to obtain a third vector.

[0096] The third vector is a vector with rich features obtained by concatenating entities with high similarity contained in the image data and the text data.

[0097] In one embodiment, the third similarity is obtained by calculating the Euclidean distance, the Manhattan distance, or the Chebyshev distance between the first vector and the second vector.

[0098] In another embodiment, the third similarity is obtained by calculating the cosine similarity between the first vector and the second vector.

[0099] In step 209, a fourth vector corresponding to the known entity is obtained, and the third vector and the fourth vector are concatenated and input into a relationship prediction model to obtain a relationship between the entities corresponding to the third vector and the fourth vector.

[0100] Specifically, the fourth vector can be obtained by encoding the known entity using an encoder. The encoder can be any encoder, for example, a BERT encoder or a BiLSTM encoder.

[0101] The relationship prediction model is composed of a multilayer perceptron (MLP) and a softmax function. The relationships corresponding to the first k items of the softmax function output, sorted from largest to smallest, are selected as the relationships between the entities corresponding to the third vector and the fourth vector.

[0102] In a specific embodiment, based on the relationship between the entities corresponding to the third vector and the fourth vector, it is determined whether the aforementioned Bin-num occurs at the defect detection level or at the electrical test level.

[0103] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0104] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program instructing the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0105] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal relationship discovery method in the semiconductor field, comprising: Acquiring image data and text data, wherein the image data and text data are data related to defect detection in a semiconductor manufacturing process; generating a scene graph based on the image data, wherein the scene graph includes entities in the image and relationships between the entities; Generating a dependency syntax tree according to the text data, wherein the dependency syntax tree includes entities in the text and relationships between entities; Encode the text data using an encoder to obtain a text encoding vector corresponding to an entity in the text data; Generate corresponding visual semantic vectors according to the entity nodes in the scene graph; Fusing the scene graph and the dependency syntax tree according to the similarity between entities to obtain a first association graph; Concatenating the visual semantic vector and the text encoding vector according to similarity to obtain a first vector; Concatenate the first vector and the second vector corresponding to the entity in the first association graph according to similarity to obtain a third vector; A fourth vector corresponding to a known entity is obtained, and the third vector and the fourth vector are concatenated and input into a relationship prediction model to obtain a relationship between the entities corresponding to the third vector and the fourth vector.

2. The method according to claim 1, characterized in that The image data includes multiple of the following: a wafer image with defect category labels, a wafer defect detection image, a photo of a wafer defect, and an electrical test result image; The text data includes multiple of the following: semiconductor equipment user manuals, semiconductor equipment troubleshooting manuals, and fault analysis documents written by semiconductor engineers; The entities in the image and / or text include multiple of the following: equipment for producing wafers, wafers, sensors, die failure categories, wafer defect categories, wafer defect coordinates, and production failure categories.

3. The method according to claim 1, characterized in that The scene graph and the dependency syntax tree are fused according to the similarity between entities to obtain a first association graph, including: Calculating a first similarity between a first entity in the scene graph and a second entity in the dependency syntax tree based on a first matrix representation corresponding to the scene graph and a second matrix representation corresponding to the dependency syntax tree; When the first similarity is greater than a preset first threshold, the first entity and the second entity are fused to obtain a first association graph.

4. The method according to claim 1, wherein The visual semantic vector and the text encoding vector are concatenated according to similarity to obtain a first vector, including: Calculating a second similarity between the visual semantic vector and the text encoding vector; When the second similarity is greater than a preset second threshold, the visual semantic vector and the text encoding vector are concatenated to obtain a first vector.

5. The method according to claim 4, characterized in that The method for calculating the second similarity includes: Calculating the Euclidean distance between the visual semantic vector and the text encoding vector to obtain a second similarity; or Calculating the Manhattan distance between the visual semantic vector and the text encoding vector to obtain a second similarity; or Calculating a Chebyshev distance between the visual semantic vector and the text encoding vector to obtain a second similarity; or The cosine similarity between the visual semantic vector and the text encoding vector is calculated to obtain a second similarity.

6. The method according to claim 1, characterized in that The first vector is concatenated with the second vector corresponding to the entity in the first association graph according to similarity to obtain a third vector, including: calculating a third similarity between the first vector and the second vector; When the third similarity is greater than a preset third threshold, the first vector and the second vector are concatenated to obtain a third vector.

7. The method according to claim 6, characterized in that The method for calculating the third similarity includes: Calculating the Euclidean distance between the first vector and the second vector to obtain a third similarity; or Calculating the Manhattan distance between the first vector and the second vector to obtain a third similarity; or calculating a Chebyshev distance between the first vector and the second vector to obtain a third similarity; or Calculate the cosine similarity between the first vector and the second vector to obtain a third similarity.

8. The method according to claim 1, characterized in that Generating a scene graph according to the image data, comprising: The image data is input into a regional convolutional neural network to obtain a scene graph.

9. The method according to claim 1, characterized in that The encoder is a BERT encoder or a BiLSTM encoder.

10. The method according to claim 1, characterized in that The relationship prediction model is composed of a multi-layer perceptron and a Softmax function.

Images