A superconducting material prediction method and device based on a pre-trained deep neural network

Abstract

The application discloses a superconducting material prediction method based on a pre-training deep neural network, which fully utilizes a large amount of unlabeled data, pre-trains through a self-supervised learning mode of atomic masking to obtain a second embedding layer, a second fusion layer and a second attention layer, constructs a superconducting material prediction model through the above structure, trains through a loss function constructed by a small amount of real superconducting material labels and a prediction superconducting material result, and obtains a superconducting material prediction model with good robustness and capable of accurately judging whether a crystal is a superconducting material.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of superconducting materials, and particularly relates to a superconducting material prediction method and device based on a pre-trained deep neural network. BACKGROUND

[0002] Superconductors are essential for sustainable daily applications and are also very attractive for the United Nations' Sustainable Development Goals (SDGs). Superconductors have zero resistance, which allows perfect transmission of current without any loss. With superconducting cables, remote areas can have a continuous power supply to provide irrigation water in time, which is a key to the sustainability of agriculture. Affordable clean energy can also be well obtained, because direct current power from places with sufficient power can be delivered to all parts of the world without loss. Zero resistance means "no heat and no loss of energy", which is a key to climate action. For human health and well-being, superconducting magnets can produce high-quality, high-intensity magnetic fields, and are well applied as high-field magnets in magnetic resonance imaging, nuclear magnetic resonance, water purification, magnetic drug delivery, etc.

[0003] Superconductivity is of great significance to human sustainable development goals, and the research and development of superconducting materials also appears in many major plans of China's development as an important high-tech. However, the theory of superconductivity is still a mystery. Benefiting from a large number of open data sets in the field of material science, many machine learning methods have been applied to predict material properties, generate structures, etc. This also provides an inspiration for modeling superconductors with machine learning methods.

[0004] A method for predicting new superconducting materials based on a deep neural network algorithm is disclosed in Chinese patent No. CN111798940A. The method uses a deep neural network to predict the performance of superconducting materials, uses a machine learning method based on the big data of existing materials and the deep neural network, and optimizes the feature quantities related to the superconducting temperature by using a genetic algorithm. More rapidly optimize training error and test error, improve prediction accuracy, can process a large amount of sample data, and better predict the superconducting performance of unknown materials, and provide a basis for experiments and material selection.

[0005] The above machine learning method mainly predicts superconductivity through the measured properties of the material, and these properties are obtained through complex and expensive feature engineering. In the cross-research of physical science and artificial intelligence, people have made a lot of efforts in modeling molecules with graph neural networks (GNN) because molecules are natural graphs. However, there are relatively few methods proposed for crystal modeling. Due to the different labels of different data sets, the current supervised learning model can only be trained on a small data set, while other unlabeled data is not fully utilized, that is, there is a problem of superconducting data shortage.

[0006] Therefore, it is urgent to develop a prediction method and device capable of accurately determining whether a crystal is a superconducting material. SUMMARY

[0007] The application provides a superconducting material prediction method based on a pre-trained deep neural network, which can obtain a superconducting material prediction model through training of a large amount of data, and accurately determine whether an input crystal is a superconducting material by using the model.

[0008] A superconducting material prediction method based on a pre-trained deep neural network comprises the following steps:

[0009] (1) Construct a pre-training sample set, obtain the position coordinates of each atom in a crystal and the element category true value label, mask part of the atomic number set in the crystal to obtain a masked crystal, the masked crystal comprises a first atomic number set and a first atomic position set, and the plurality of masked crystals are used as the pre-training sample set;

[0010] (2) Construct a pre-training model, the pre-training model comprises a first embedding layer, a first fusion layer, a first attention layer and a classifier layer, wherein the first embedding layer is used for embedding the first atomic number set into a multidimensional vector to obtain a first multidimensional vector, the Euclidean distance between each atomic position in the first atomic position set and the center position of the crystal is used to obtain an atomic coordinate position vector by adopting Gaussian operation; the first fusion layer is used for fusing each atomic number in the first multidimensional vector with the corresponding atomic coordinate position to obtain a first fusion vector composed of atomic fusion data of each atom; the first attention layer is used for fusing the fusion data of each atom and its surrounding atoms again to obtain atomic attention fusion data of each atom, and a plurality of atomic attention fusion data construct a first attention vector, the first attention vector is output to the classifier layer, and the class prediction value of the masked atom is obtained through the full connection network in the classifier layer;

[0011] (3) Train the pre-training model through the pre-training sample set, input the class prediction value of the masked atom and the true value label of the masked atom into a first cross-entropy loss function, optimize the loss, and obtain a pre-trained deep neural network with determined parameters after reaching a loss threshold, and freeze the second embedding layer, the second fusion layer and the second attention layer in the pre-trained deep neural network, wherein the second embedding layer, the second fusion layer and the second attention layer are obtained after training the first embedding layer, the first fusion layer and the first attention layer in the pre-training model respectively;

[0012] (4) Construct a training sample set, obtain the superconductor category true value label of each crystal, and construct a training sample set with a plurality of crystals, each crystal comprising a second atomic number set, a second atomic position set and a lattice parameter set;

[0013] (5) Construct a training model, which includes a second embedding layer, a second fusion layer, a second attention layer, a Set2Set layer, a third fusion layer, and a classifier layer; input the second atomic number set and the second atomic position set into the second embedding layer, the second fusion layer, and the second attention layer to obtain the second attention vector; output the second attention vector into the Set2Set layer to obtain the representation vector; output the representation vector and the lattice parameter set into the third fusion layer to obtain the fused representation vector; output the fused representation vector into the classifier to obtain the predicted value of the crystal superconductor;

[0014] (6) The training model is trained by training sample set. The predicted value of the crystal superconductor category and the true value label of the superconductor category are output to the second cross-entropy loss function for training. The model parameters are determined to obtain the superconductor prediction model.

[0015] The atomic coordinate position vector is obtained by performing Gaussian operations based on the Euclidean distance between each atomic position in the first atomic position set and the crystal center position, including:

[0016] The set of Euclidean distances between each atomic position in the first atomic position set and the crystal center is taken as the first Euclidean distance set. The set of Euclidean distances between each atomic position in the first atomic position set and the origin is taken as the second Euclidean distance set. Gaussian operations are performed on the first Euclidean distance set and the corresponding second Euclidean distance set to obtain the atomic coordinate position vector in the masked crystal.

[0017] Gaussian operations are performed on the set of the first Euclidean distances and the corresponding set of the second Euclidean distances to obtain the atomic coordinate position vectors in the shielded crystal. for:

[0018]

[0019] Where σ is a hyperparameter, p' is the set of first Euclidean distances, and p' o For the set of the second Euclidean distance.

[0020] The first fusion vector A”, composed of the fusion data of each atom, is:

[0021]

[0022] Where A' is the first multidimensional vector, W a The first-dimensional parameter matrix is ​​used to transform the dimension of the atomic coordinate position vector to be the same as the dimension of the first multi-dimensional vector. The fused data is a". i .

[0023] The first attention layer is used to perform attention fusion on the fusion data of each atom and its surrounding atoms to obtain attention fusion data for each atom. The specific steps of attention fusion are as follows:

[0024] The first attention parameter vector of the first attention layer is W. Q W K and W V A distance threshold is set, and neighboring atoms centered on the i-th atom that satisfy the distance threshold are obtained. A second fusion vector is constructed using the fusion data of the i-th atom and its neighboring atoms. This second fusion vector is then reshaped from a d1-dimensional vector into a two-dimensional tensor of shape (d2, d1 / d2), and then compared with W... Q W K and W V Multiplying them yields the second attention parameter matrix {Q, K, V}, where W Q W K and W V The shape of each atom is (d2, d1 / d2, d1 / d2). Based on the second attention parameter matrix, the fusion data of the i-th atom and its neighboring atoms are fused by attention to obtain the attention fusion data of the i-th atom. Then, the fusion data is reshaped from a two-dimensional tensor of shape (d2, d1 / d2) into a single d1-dimensional vector.

[0025] The i-th atomic attention fusion data a”' i for:

[0026]

[0027]

[0028] Where softmax(·) is the activation function, Norm(·) is the layer normalization, and K... T The scores are the transpose of the second attention parameter matrix K. i,j It is a vector score i The j-th component, v j Let d be the j-th vector in the second attention parameter matrix V, where N is the number of the central atom and its neighboring atoms. qi For vector q i The dimension, a” i Let A be the i-th vector.

[0029] The first attention fusion vector is then subjected to attention fusion again, and this process is repeated multiple times to obtain the final attention fusion vector.

[0030] First cross-entropy loss function for:

[0031]

[0032] Where n is the number of atoms, i is the atom index in the shielded crystal, M is the number of element categories, t is the element category index, and y i,t Let the i-th atom be the true value of element category t. Let be the predicted value for the i-th atom as element category t.

[0033] The fused characterization vector b' is obtained by outputting the characterization vector and lattice parameter set to the third fusion layer:

[0034] b' = b + W θ *U

[0035] Where b is the representation vector, W θ The second-dimensional parameter matrix is ​​used to transform the dimension of the lattice parameter U to be the same as the dimension of the characterization vector b.

[0036] Second cross-entropy loss function for:

[0037]

[0038] Where L is the number of crystals, y l The true value label for the l-th crystal is a superconducting material. The predicted value is that the l-th crystal is a superconducting material.

[0039] A superconducting material prediction device based on a pre-trained deep neural network includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. The computer memory also stores a superconductor prediction model, which is constructed using a superconducting material prediction method based on a pre-trained deep neural network. When the computer processor executes the computer program, it performs the following steps:

[0040] The atomic number, atomic position, and lattice parameters of a crystal are input into a superconductor prediction model to predict whether the crystal is a superconductor.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] (1) This invention proposes a pre-trained deep neural network model from structure to superconductivity (S2SNet), which only requires the crystal structure to model the interactions between atoms in the lattice to predict the superconducting properties of the material. An attention mechanism is applied to the crystal structure. S2SNet is pre-trained on a large amount of unlabeled data to learn the interactions between different atoms in the lattice.

[0043] (2) This invention firstly makes full use of a large amount of unlabeled data, and pre-trains it through a self-supervised learning method of atomic masking to obtain a second embedding layer, a second fusion layer and a second attention layer. Then, it constructs a superconducting material prediction model through the above structure, and trains it through a loss function constructed by a small number of real superconducting material labels and the predicted superconducting material results, so as to obtain a robust superconducting material prediction model that can accurately judge whether a crystal is a superconducting material. Attached Figure Description

[0044] Figure 1 A schematic diagram illustrating a method for predicting superconducting materials based on a pre-trained deep neural network, provided by this invention;

[0045] Figure 2 A block diagram of a pre-trained deep neural network provided in Example 1;

[0046] Figure 3 The flowchart of the attention layer provided in Example 1 is as follows, with "atom a1 as the center and a2 and a3 as neighbors".

[0047] Figure 4 The block diagram of a classifier for a superconducting material prediction method based on a pre-trained deep neural network is provided in Example 1;

[0048] Figure 5 This is a block diagram of a superconducting material prediction method based on a pre-trained deep neural network, as provided in Example 1. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0050] This invention first masks some atoms in a large amount of unlabeled crystal data and pre-trains a model by predicting the elemental types of the masked atoms. Partial structures in the pre-trained model are then frozen, and a Set2Set layer and a classification layer are added to construct a new model. Using small-sample superconducting data and superconducting labels, a model is trained to predict whether a crystal is a superconducting material, thereby screening out materials with superconducting properties, such as… Figure 1 As shown.

[0051] Example 1

[0052] This invention provides a method for predicting superconducting materials based on a pre-trained deep neural network, comprising:

[0053] (1) Multiple crystals were obtained. The data for the superconducting material came from the superconducting material database of the National Institute for Materials Science, Japan, and the corresponding structural information was found in the Material Project to establish a superconducting sample dataset. Each crystal c is composed of the second atomic number set A = {a1, a2, ..., a...} n The second set of atomic positions, P = {p1, p2, ..., p...} n} and lattice parameters U = {u1, u2, ..., u n The composition is c = {A, P, U}, where a i p i u i Let represent the atomic number, actual three-dimensional position, and physical size of the lattice of the i-th atom, respectively, and n be the number of atoms in the crystal.

[0054] Obtain label-free material crystal data, which comes from the Material Project database. Obtain the position coordinates of each atom in the crystal to construct the first atom position set, and obtain the true value labels of 95 common element categories for each atom. Mask the ordinal numbers of 15% (at least 2 atoms) of atoms in the crystal, replace the masked atoms with masks to obtain masked crystals, and construct the corresponding first atom ordinal number set. Use multiple masked crystals as a pre-training sample set to pre-train the neural network.

[0055] (2) Construct a pre-trained model, such as Figure 2 As shown, the specific steps are as follows:

[0056] (2.1) The pre-trained model includes a first embedding layer, a first fusion layer, a first attention layer, and a classifier layer. The first embedding layer embeds the first atomic number set A* into a 512-dimensional vector to obtain a first multi-dimensional vector. The Gaussian expansion distance centered at the 512 points is applied to the positions P of the atoms in the crystal, where these 512 points are linearly placed between 0 and... Between these two sets, the Euclidean distances between each atomic position in the first atomic position set and the crystal center are combined to form the first Euclidean distance set. The Euclidean distances between each atomic position in the first atomic position set and the origin are combined to form the second Euclidean distance set. A Gaussian operation is then performed on the first and second Euclidean distance sets to obtain the atomic coordinate position vectors in the masked crystal. for:

[0057]

[0058] Where σ is a hyperparameter, p' is the set of first Euclidean distances, and p' o The set of the second Euclidean distance, i.e., from 0 to... There are 512 distances, and σ = 2.

[0059] The first fusion layer is used to fuse each atomic number in the first multidimensional vector with its corresponding atomic coordinate position to obtain a first fusion vector A” composed of the fused data of each atom:

[0060]

[0061] Where A' is the first multidimensional vector, W a The first-dimensional parameter matrix is ​​used to transform the dimension of the atomic coordinate position vector to be the same as the dimension of the first multidimensional vector.

[0062] (2.2) The first attention layer is used to fuse the fusion data of each atom and its surrounding atoms again. This fusion is repeated 10 times to obtain the attention fusion data of each atom. The attention fusion data of multiple atoms construct the first attention vector. The specific steps of each fusion are as follows:

[0063] The first attention parameter vector of the first attention layer is W. Q W K and W V Set the Euclidean distance threshold Using fused data as a” i Taking the atom as the center, obtain the neighboring atoms that satisfy the Euclidean distance threshold. The fused data of the neighboring atoms are {a” i+1 ,a” i+2 ,…,a” i+n-1}, using a” i and {a” i+1 ,a” i+2 ,…,a” i+n-1 Construct a second fusion vector, then reshape it from a 512-dimensional vector into a 2D tensor of shape (8, 64). Connect this tensor to matrix W through fully connected layers. Q W K and W V Multiplying them yields the second attention parameter matrix {Q, K, V}, where Q = {q i ,q i+1 ,q i+2 ,…,q i+n-1}, K = {k i ,k i+1 ,k i+2 ,…,k i+n-1}, V={v i ,v i+1 ,v i+2 ,…,v i+n-1}:

[0064] q i =a”i *W Q

[0065] k i =a” i *W K

[0066] v i =a” i *W V

[0067] Among them W Q W K and W V The shapes are all (8, 64, 64).

[0068] Based on the second attention parameter matrix, the attention mechanism is used to focus on atom a” i and its neighboring atoms {a” i+1 ,a” i+2 ,…,a” i+n-1 The initial atomic attention fusion data a”' is obtained by performing attention fusion on the fused data. i° The specific steps are as follows:

[0069]

[0070]

[0071] a” i ≡a”' i°

[0072] Update a” i For a”' i° Then, attention fusion is performed again, and after S iterations, a”' is obtained. i Where softmax(·) is the activation function, Norm(·) is the layer normalization, and K... T The scores are the transpose of the second attention parameter matrix K. i,j It is a vector score i The j-th component, v j Let d be the j-th vector in the second attention parameter matrix V, N be the number of the i-th atom and its neighboring atoms, and d be the vector of the i-th atom. qi For vector q i The dimension, a” i Let A be the i-th vector.

[0073] The data will then be reshaped to merge the data a”' i The two-dimensional tensor of shape (8, 64) is transformed into a single 512-dimensional vector.

[0074] like Figure 3As shown, when i = 1, the neighboring atoms within the Euclidean threshold range are two atoms, i.e., a”2, a”3. The second attention parameter matrix {Q,K,V}, Q = {q1,q2,q3}, K = {k1,k2,k3}, V = {v1,v2,v3}, yields the intermediate attention fusion data z”'1 as follows:

[0075]

[0076]

[0077] (2.2) Output the first attention vector to the classifier layer, and use the two fully connected layers in the classifier layer to classify 95 common elements to obtain the category prediction value of the masked atom.

[0078] (3) A pre-trained model is trained using a pre-trained sample set. The predicted class values ​​and true labels of the masked atoms are input into the first cross-entropy loss function. The model is then optimized using the stochastic gradient descent algorithm to obtain a pre-trained deep neural network with determined parameters. The pre-trained deep neural network includes a first embedding layer, a first fusion layer, a first attention layer, and a classification layer. The first embedding layer, the first fusion layer, and the first attention layer in the pre-trained deep neural network with determined parameters are frozen and used as the second embedding layer, the second fusion layer, and the second attention layer, respectively.

[0079] Wherein, the first cross-entropy loss function for:

[0080]

[0081] Where n is the number of atoms, i is the atom index in the shielded crystal, M is the number of element categories, t is the element category index, and y i,t Let the i-th atom be the true value of element category t. Let be the predicted value for the i-th atom as element category t.

[0082] (4) Construct a training sample set and obtain the true value label of the superconductor category for each crystal. Construct training samples for multiple crystals in step (1). Each crystal includes a second atomic number set A, a second atomic position set P, and a lattice parameter set U.

[0083] (5) Construct a training model, such as Figure 4As shown, the training model includes a second embedding layer, a second fusion layer, a second attention layer, a Set2Set layer, a third fusion layer, and a classifier. The second atomic number set A and the second atomic position set P are sequentially input into the second embedding layer, the second fusion layer, and the second attention layer to obtain the second attention vector. The second attention vector is output to the Set2Set layer to obtain the representation vector. The representation vector and the lattice parameter set are output to the third fusion layer to obtain the fused representation vector b'.

[0084] b' = b + W θ *U

[0085] Where b is the representation vector, W θ The second-dimensional parameter matrix is ​​used to transform the dimension of the lattice parameter U to be the same as the dimension of the characterization vector b.

[0086] The fused representation vector is output to the classifier to obtain the predicted value of the crystalline superconductor, such as... Figure 5 As shown, the classifier consists of a fully connected network. The model is trained by training the training sample set, and the predicted value of the crystal superconductor category and the true value label of the superconductor category are output to the second cross-entropy loss function for training. The model parameters are then determined to obtain the superconductor prediction model.

[0087] Wherein, the second cross-entropy loss function for:

[0088]

[0089] Where L is the number of crystals, y l The true value label for the l-th crystal is a superconducting material. The predicted value is that the l-th crystal is a superconducting material.

[0090] A device for predicting superconducting materials based on a pre-trained deep neural network includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. The computer memory also stores a superconductor prediction model (S2SNet), which is constructed using a superconducting material prediction method based on a pre-trained deep neural network. When the computer processor executes the computer program, it performs the following steps:

[0091] The atomic number, atomic position, and lattice parameters of a crystal are input into a superconductor prediction model to predict whether the crystal is a superconductor.

[0092] (1) The method for predicting superconducting materials based on pre-trained deep neural network algorithm provided by the present invention pre-trains a deep neural network using a large amount of existing label-free data, and then fine-tunes the neural network using superconducting material data. It has a large sample processing capacity and accurate prediction results.

[0093] (2) The method for predicting superconducting materials based on a pre-trained deep neural network algorithm provided in this invention predicts whether a material has superconducting properties starting from the structure of the crystal material. It makes full use of the structural information of the crystal material, including atomic coordinates and their indices, and lattice parameters. It avoids manual feature engineering and has higher prediction efficiency. Existing algorithms almost entirely rely on traditional machine learning methods and manually designed features, which are computationally intensive, time-consuming, and have poor results.

[0094] Comparative Example 1

[0095] This comparative example provides a method for predicting superconducting materials based on a graph message propagation network algorithm. The method adopts the graph message propagation network method described in "Graph networks as a universal machine learning framework for molecules and crystals", Chi Chen, et al. Chemistry of Materials, 31(9):3564–3572, 2019.

[0096] Comparative Example 2

[0097] This comparative example provides a method for predicting superconducting materials based on the random forest algorithm. The method uses the random forest method described in "Machine learning modeling of superconducting critical temperature", Valentin Stanev, et al. NPJ Computational Materials, (2018) 4:29.

[0098] Comparative Example 3

[0099] This comparative example provides a method for predicting superconducting materials based on a graph convolutional network algorithm. The method adopts the crystal graph convolutional network method described in "Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.", Tian Xie and Jeffrey C. G. Rossman. Physical Review Letters, 120(14):145301, 2018.

[0100] Performance testing:

[0101] Using the methods provided in the examples and comparisons, sample training was performed. Accuracy (ACC) and Area Under the Curve (AUC) were used to characterize the training performance of the machine learning model, and the accuracy was plotted in a table, as shown in Table 1. Table 1 shows that S2SNet consistently outperforms its competitors in both ACC and AUC. For example, without pre-training, S2SNet improves ACC by approximately 6% over Statistic, AUC by approximately 7% over CGCNN, and ACC / AUC by nearly 11% / 14% over MEGNet. Through pre-training tasks, all three models—MEGNet, CGCNN, and S2SNet—show significant improvements. Because S2SNet is pre-trained on unsupervised MLM tasks, it can be pre-trained on larger datasets without any labels. However, MEGNet and CGCNN must be pre-trained on datasets with specific labels, such as forming energies. This also means that S2SNet can be further improved with larger datasets in future work.

[0102] Meanwhile, this invention trains the machine to ACC > 91% using the method provided in the embodiments, and then predicts all materials in MaterialProject. The prediction results are shown in Table 2. * indicates that the superconductor has been discovered. We present a series of high-scoring superconducting materials for experimental physicists to verify. As can be seen from Table 2, the method provided by this invention can predict the superconductivity probability of different materials, thus providing better guidance for experiments and enabling the screening of materials with superconducting potential, providing a good foundation for material screening and experimentation.

[0103] In summary, the method for predicting superconducting materials based on a pre-trained deep neural network algorithm provided by this invention pre-trains the model using a large amount of existing label-free data, and then fine-tunes the model using scarce superconducting materials. Compared to traditional manual feature engineering, this method predicts whether a material possesses superconducting properties starting from the structure of the crystal material, making full use of the structural information of the crystal material, including atomic coordinates and their indices, and lattice parameters. It avoids manual feature engineering, resulting in higher prediction efficiency and accuracy; compared to existing crystal model neural networks, it makes better use of label-free data, achieving even higher accuracy.

[0104] Table 1. Accuracy Prediction Data for Superconducting Materials

[0105]

[0106] Table 2. Prediction of superconductivity probability for different materials

[0107]

Claims

1. A method for predicting superconducting materials based on a pre-trained deep neural network, characterized in that, include: (1) Construct a pre-training sample set, obtain the position coordinates of each atom in the crystal and the true value label of the element category, mask part of the atomic number set in the crystal to obtain a masked crystal, the masked crystal includes the first atomic number set and the first atomic position set, and use multiple masked crystals as a pre-training sample set; (2) Construct a pre-trained model. The pre-trained model includes a first embedding layer, a first fusion layer, a first attention layer, and a classifier. The first embedding layer is used to embed the first atomic number set into a multi-dimensional vector to obtain the first multi-dimensional vector. The atomic coordinate position vector is obtained by Gaussian operation based on the Euclidean distance between each atomic position in the first atomic position set and the crystal center position. The first fusion layer is used to fuse each atomic number in the first multi-dimensional vector with the corresponding atomic coordinate position to obtain the first fusion vector composed of the fusion data of each atom. The first attention layer is used to fuse the fusion data of each atom and its surrounding atoms again to obtain the attention fusion data of each atom. The attention fusion data of multiple atoms constructs the first attention vector. The first attention vector is output to the classifier. The class prediction value of the occluded atom is obtained through the fully connected network in the classifier. (3) Train the pre-trained model through the pre-trained sample set, input the predicted class value of the masked atom and the real value label of the masked atom into the first cross-entropy loss function, optimize the loss, and obtain the pre-trained deep neural network with determined parameters after reaching the loss threshold. Freeze the second embedding layer, the second fusion layer and the second attention layer in the pre-trained deep neural network, wherein the second embedding layer, the second fusion layer and the second attention layer are obtained after training the first embedding layer, the first fusion layer and the first attention layer in the pre-trained model, respectively. (4) Construct a training sample set and obtain the true value label of the superconductor category for each crystal. Construct training samples from multiple crystals. Each crystal includes a second atomic number set, a second atomic position set, and a lattice parameter set. (5) Construct a training model, which includes a second embedding layer, a second fusion layer, a second attention layer, a Set2Set layer, a third fusion layer, and a classifier layer; input the second atomic number set and the second atomic position set into the second embedding layer, the second fusion layer, and the second attention layer to obtain the second attention vector; output the second attention vector into the Set2Set layer to obtain the representation vector; output the representation vector and the lattice parameter set into the third fusion layer to obtain the fused representation vector; output the fused representation vector into the classifier to obtain the predicted value of the crystal superconductor; (6) The training model is trained by training sample set. The predicted value of the crystal superconductor category and the true value label of the superconductor category are output to the second cross-entropy loss function for training. The model parameters are determined to obtain the superconductor prediction model.

2. The method for predicting superconducting materials based on a pre-trained deep neural network according to claim 1, characterized in that, The atomic coordinate position vector is obtained by performing Gaussian operations based on the Euclidean distance between each atomic position in the first atomic position set and the crystal center position, including: The set of Euclidean distances between each atomic position in the first atomic position set and the crystal center is taken as the first Euclidean distance set. The set of Euclidean distances between each atomic position in the first atomic position set and the origin is taken as the second Euclidean distance set. Gaussian operations are performed on the first Euclidean distance set and the corresponding second Euclidean distance set to obtain the atomic coordinate position vector in the masked crystal.

3. The method for predicting superconducting materials based on a pre-trained deep neural network according to claim 2, characterized in that, Gaussian operations are performed on the set of the first Euclidean distances and the corresponding set of the second Euclidean distances to obtain the atomic coordinate position vectors in the shielded crystal. for: ,in, For hyperparameters, For the set of the first Euclidean distance. For the set of the second Euclidean distance.

4. The method for predicting superconducting materials based on a pre-trained deep neural network according to claim 1, characterized in that, The first fusion vector is composed of the fusion data of each atom. for: in, For the first multidimensional vector, The first-dimensional parameter matrix is ​​used to transform the dimension of the atomic coordinate position vector to be the same as the dimension of the first multidimensional vector.

5. The method for predicting superconducting materials based on a pre-trained deep neural network according to claim 1, characterized in that, The first attention layer is used to fuse the fusion data of each atom and its surrounding atoms again to obtain the attention fusion data of each atom, including: the first attention parameter vector of the first attention layer is W. Q W K and W V Set a distance threshold and obtain the first... i Taking the nth atom as the center, obtain neighboring atoms that satisfy the distance threshold, and use the nth atom... i The second fusion vector is constructed from the fused data of 1 atom and its neighboring atoms. The second fusion vector is then reshaped from... d 1 A dimensional vector is transformed into a vector of shape ( d 2 , d 1 / d 2 The two-dimensional tensor of ) is respectively related to W Q W K and W V Multiplication yields the second attention parameter matrix. W Q W K and W V The shapes are all ( d 2 , d 1 / d 2 , d 1 / d 2 Based on the second attention parameter matrix, the first... i Attention fusion is performed on the fused data of the _th atom and its neighboring atoms to obtain the _th _th atom. i The attention fusion data is composed of individual atoms, and then reshaped to change the shape of the attention fusion data from ( d 2 , d 1 / d 2 The two-dimensional tensor becomes a single d 1 The first attention fusion vector of dimension.

6. The method for predicting superconducting materials based on a pre-trained deep neural network according to claim 5, characterized in that, No. i Atom attention fusion data for: in, For activation function, For layer normalization, The second attention parameter matrix transpose, It is a vector The j One portion, The second attention parameter matrix The first in j A vector, N The number of the central atom and its neighboring atoms. For vectors Dimensions for The A vector.

7. The method for predicting superconducting materials based on a pre-trained deep neural network according to claim 5 or 6, characterized in that, The first attention fusion vector is then subjected to attention fusion again, and this process is repeated multiple times to obtain the final attention fusion vector.

8. The method for predicting superconducting materials based on a pre-trained deep neural network according to claim 1, characterized in that, The characterization vector and lattice parameter set are output to the third fusion layer to obtain the fused characterization vector. for: in, b For the representation vector, This is the second-dimensional parameter matrix, used to transform the lattice parameters. U Dimensions and representation vectors b They have the same dimensions.

9. The superconducting material prediction method based on a pre-trained deep neural network according to claim 1, wherein the second cross-entropy loss function... for: in, L The number of crystals, For the first l Each crystal is a true value label for superconducting materials. For the first l The predicted value for each crystal is a superconducting material.

10. A device for predicting superconducting materials based on a pre-trained deep neural network, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that, The computer memory also stores a superconductor prediction model, which is constructed using the superconducting material prediction method based on a pre-trained deep neural network as described in any one of claims 1 to 9. When the computer processor executes the computer program, it performs the following steps: The atomic number, atomic position, and lattice parameters of a crystal are input into a superconductor prediction model to predict whether the crystal is a superconductor.

Images