A small molecule chemical structure image recognition method based on a transformation neural network
By combining MobileViT_2 and Conditional DETR networks, the problems of low prediction accuracy of complex sequences, slow model convergence, and unstable weights in small molecule chemical structure image recognition are solved, achieving more efficient and accurate small molecule chemical structure image recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIHUA UNIV
- Filing Date
- 2022-11-29
- Publication Date
- 2026-05-08
AI Technical Summary
Existing technologies for small molecule chemical structure image recognition suffer from problems such as low prediction accuracy for complex sequences, slow model convergence speed, and unstable model learning. In particular, for sequences with many character types and a large number of characters, the feature extraction capability is weak, and fluctuations and weight instability are prone to occur during model training.
The MobileViT_2 network and the Conditional DETR network are combined. By preprocessing small molecule chemical structure images, feature vectors are extracted and decoded into SMILES sequences. Lightweight networks and learning rate warm-up techniques are used to adjust the model structure to improve feature extraction and model convergence speed, and stabilize the weights.
It improves the prediction accuracy of complex sequences, shortens the model training time, ensures the stability of the model and the weights, and enhances the overall recognition effect.
Smart Images
Figure CN115909328B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a method for image recognition of small molecule chemical structures based on transform neural networks. Background Technology
[0002] Small molecule drugs have a wide range of applications, including oncology, nervous system disorders, infections, metabolic disorders, cardiovascular diseases, non-central analgesia, antipyretics, anti-inflammatories, immune or allergic diseases, dermatological diseases, digestive system diseases, and bone diseases. Statistics show that small molecule drugs account for up to 98% of commonly used drugs. In the field of life science data management, extracting chemical structures from publicly available resources (such as journal articles and patents) has always been difficult and time-consuming. In recent years, with the rapid development of computer vision and natural language processing technologies based on deep learning algorithms, the use of deep learning technology to extract valuable information from images has become increasingly widespread. Deep neural networks can automatically extract features and have good robustness and generalization ability on chemical structure images. Because the structural details of small molecule compounds in many small molecule-related literature are presented in image formats such as JPEG, PNG, GIF, and BMP, they lose their original chemical meaning. Automatically analyzing these chemical structure images and converting them into computer-recognizable formats, such as SMILES notation, has practical application value for the analysis and discovery of small molecule drugs. Researchers both domestically and internationally have conducted extensive work and made significant progress in the study of image recognition of small molecule chemical structures, but the following problems still exist:
[0003] First, the prediction accuracy for complex sequences is low: the image recognition performance of small molecule chemical structures is reflected by both subjective and objective evaluation metrics. Current research combines image feature extraction networks with Transformer neural networks. This algorithm performs well in predicting simple sequences with few character types and a small number of characters. However, small molecule databases also contain many complex sequences with a large number of character types. Furthermore, because chemical structure images themselves carry limited and sparse image feature information, their feature representation ability is weak, resulting in less and less feature information that can be extracted during feature extraction. This leads to incomplete or insufficient feature information, resulting in errors in character and chemical bond recognition in subsequent predictions. Although current algorithms perform well in predicting simple sequences, their performance on complex sequences is generally poor across both subjective and objective evaluation metrics.
[0004] Secondly, the model convergence speed is slow: In the research of chemical structure image recognition algorithms, many scholars have used the classic encoder-decoder structure transform neural network model to improve the accuracy of predicted sequences, achieving good results. However, due to the high complexity and large number of parameters of the classic transform neural network model, the training process requires a large amount of data. Although the overall effect of the final sequence prediction has been improved to a certain extent, the convergence speed of the model is slow, and its practical value is low.
[0005] Third, model learning fluctuates, and the learned weights are unstable: Because the deep learning models currently used are deep and large, the variance of the data distribution within the mini-batch is particularly large during the training process, which will cause the model learning to fluctuate drastically, making the learned weights very unstable. Summary of the Invention
[0006] To address the above problems, this invention proposes a method for small molecule chemical structure image recognition based on transform neural networks.
[0007] The technical solution of this invention is: a method for image recognition of small molecule chemical structures based on transform neural networks, comprising the following steps:
[0008] S1: Acquire images of small molecule chemical structures and preprocess them;
[0009] S2: Use the preprocessed small molecule chemical structure image as input to the MobileViT_2 network to extract the feature vector of the small molecule chemical structure image;
[0010] S3: Obtain the original SMILES sequence, use it as a label, and combine it with the feature vector of the small molecule chemical structure image as input to the decoding part of the Conditional DETR network to obtain the SELFIES sequence. Then, use the selfies package to convert it into a new SMILES sequence as the recognition result to complete the recognition of the small molecule chemical structure image.
[0011] Furthermore, in step S1, the specific method for preprocessing the small molecule chemical structure image is as follows: convert the small molecule chemical structure image into an RGB image and then randomly rotate it.
[0012] Further, in step S2, the MobileViT_2 network includes a first convolutional module, a first MobileV2block module, a second MobileV2 block module, a third MobileV2 block module, a fourth MobileV2block module, a fifth MobileV2 block module, a first MobileViT_2block module, a sixth MobileV2 block module, a second MobileViT_2block module, a seventh MobileV2 block module, a third MobileViT_2block module, a second convolutional module, and an average pooling module, which are connected in sequence.
[0013] Furthermore, the first MobileV2 block module, the second MobileV2 block module, the third MobileV2 block module, the fourth MobileV2 block module, and the fifth MobileV2 block module are used to extract feature information from the small molecule chemical structure image; among them, the second MobileV2 block module and the fifth MobileV2 block module are also used to downsample the small molecule chemical structure image.
[0014] Furthermore, the first MobileViT_2block module, the second MobileViT_2block module, and the third MobileViT_2block module have the same structure, each including a first convolutional layer, a second convolutional layer, an unfolded layer, a first set of normalized layers, a LinearAttnFFN layer, a second set of normalized layers, a folded layer, a third convolutional layer, and a fourth convolutional layer connected in sequence.
[0015] Furthermore, the first MobileV2 block module, the second MobileV2 block module, the third MobileV2 block module, the fourth MobileV2 block module, the fifth MobileV2 block module, the sixth MobileV2 block module, and the seventh MobileV2 block module have the same structure, including a convolutional layer, a SiLU activation layer, a normalization layer, and a deformable convolutional block connected in sequence.
[0016] Furthermore, the decoding part of the Conditional DETR network includes a first sublayer connection module, a second sublayer connection module, and a third sublayer connection module;
[0017] The original SMILES sequence is used as the input to the first and second sub-layer connection modules. The first and second sub-layer connection modules are used to normalize, learn features, and concatenate the original SMILES sequence. The third sub-layer connection module is used to decode the processing result of the second sub-layer connection module to generate SEFILES format characters, obtain the SELFIES sequence, and convert it into a new SMILES sequence through the selfies package.
[0018] Furthermore, the first sub-layer connection module includes a first normalized layer, a first multi-head self-attention sub-layer, and a first residual layer; the second sub-layer connection module includes a second normalized layer, a second multi-head attention sub-layer, and a second residual layer; the third sub-layer connection module includes a third normalized layer, a feedforward fully connected sub-layer, and a third residual layer.
[0019] The beneficial effects of this invention are:
[0020] (1) This invention combines the decoding part of MobileViT_2 with Conditional DETR to generate the chemical character representation format SMILES sequence based on chemical structure images. By using MobileViT_2 with the fully connected layers removed as the encoding part, and replacing the depthwise separable convolution in the MobileV2 module of this model with deformable convolution, more and more accurate global features of chemical structure images are obtained, improving the accuracy of prediction results for complex sequences. In addition, by using the lightweight network MobileViT_2 as the encoding part, the number of model parameters is greatly reduced compared to other models;
[0021] (2) This invention selects the decoding part of Conditional DETR as the decoding part of the overall encoder-decoder network, processes the encoded result and the input sequence, and improves the convergence speed of the model; and advances the position of the normalization layer in this part, so that the trained model performs better and is more stable, and can also effectively prevent gradient explosion or gradient vanishing of the model.
[0022] (3) The present invention selects the learning rate preheating method to adjust the learning rate during the training process, and the learned weights are more stable, which makes the model converge faster. Attached Figure Description
[0023] Figure 1 A flowchart for a method for image recognition of small molecule chemical structures;
[0024] Figure 2 This is a structural diagram of the MobileViT_2 network;
[0025] Figure 3Here is a structural diagram of the MobileViT_2Block module;
[0026] Figure 4 Here is the structure diagram of LinearAttnFFN;
[0027] Figure 5 This is a structural diagram of the MobileV2 Block;
[0028] Figure 6 This is a structural diagram of the decoding part of the Conditional DETR model;
[0029] Figure 7 These are comparison diagrams showing the effects of embodiments of the present invention. Detailed Implementation
[0030] The embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0031] MobileViT_2 network: Mobile Vision Transformation Neural Network;
[0032] Conditional DETR network: a conditional detection transformation neural network.
[0033] like Figure 1 As shown, this invention provides a method for small molecule chemical structure image recognition based on transform neural networks, comprising the following steps:
[0034] S1: Acquire images of small molecule chemical structures and preprocess them;
[0035] S2: Use the preprocessed small molecule chemical structure image as input to the MobileViT_2 network to extract the feature vector of the small molecule chemical structure image;
[0036] S3: Obtain the original SMILES sequence, use it as a label, and combine it with the feature vector of the small molecule chemical structure image as input to the decoding part of the Conditional DETR network to obtain the SELFIES sequence. Then, use the selfies package to convert it into a new SMILES sequence as the recognition result to complete the recognition of the small molecule chemical structure image.
[0037] The dataset used in this invention comes from open-source websites such as PubChem and Chemble. One million SMILES sequences were randomly selected as training data, and 5000 SMILES sequences were used as test data. First, the SMILES sequences were converted into SELFIES, a 100% robust molecular string representation, and a vocabulary was generated. Then, using the RDkit package, images with a resolution of 256×256 were generated from the character sequences represented by the SMILES method. The generated image dataset was preprocessed, converting all input images to RGB images and randomly rotating them. The images were also normalized. The preprocessed images were then used as input to the MobileViT_2 network, passed through the decoding network of the Conditional DETR model, and finally output as SMILES sequences.
[0038] This invention mainly consists of a MobileViT_2 network and a Conditional DETR decoding network to generate corresponding SMILES sequences from an input small molecule chemical structure image. The process is as follows: An input small molecule chemical structure image (256×256) is preprocessed, and then the MobileViT_2 network extracts the image's feature information. The resulting feature vector is used as the input to the multi-head attention mechanism of the second sub-layer of the Conditional DETR decoding part. The input SMILES sequence is converted into a SELFIES sequence, which is then used as the input to the Conditional DETR decoding network. Finally, the results from the decoding network are classified and output as individual characters, i.e., the SELFIES sequence, which is then passed through the selfies decoder network to obtain the SMILES sequence. The entire training process uses a learning rate warm-up method to adjust the learning rate. During the warm-up period, the learning rate linearly increases from 0 to the initial learning rate in the optimizer. After the warm-up phase, a plan is created to linearly decrease the learning rate from the initial learning rate in the optimizer back to 0. The initial learning rate is 0.00005, and the AdamW optimizer is used.
[0039] In this embodiment of the invention, the specific method for preprocessing the small molecule chemical structure image in step S1 is as follows: convert the small molecule chemical structure image into an RGB image and then randomly rotate it.
[0040] In embodiments of the present invention, such as Figure 2As shown, in step S2, the MobileViT_2 network includes a first convolutional module, a first MobileV2 block module, a second MobileV2 block module, a third MobileV2 block module, a fourth MobileV2 block module, a fifth MobileV2 block module, a first MobileViT_2 block module, a sixth MobileV2 block module, a second MobileViT_2 block module, a seventh MobileV2 block module, a third MobileViT_2 block module, a second convolutional module, and an average pooling module, all connected in sequence.
[0041] The MobileViT_2 network, as the encoding part of the overall encoding-decoding process, combines the Mobilev2 block, the inverse residual module of MobileNet. By concatenating and interleaving the Mobilev2 block and the MobileViT_2 block, it achieves the interaction of local and global visual representation information. This encoding part consists of three... Figure 3 The MobileViT_2 module shown, 7 such as Figure 5 The feature extraction network shown consists of a MobileV2 block, two convolutional modules, and one average pooling layer. Figure 4 As shown, the number of LinearAttnFFN blocks L stacked in the three MobileViT_2blocks are 1, 3, and 2, respectively. Each MobileViT_2block includes a convolutional layer, a SiLU activation layer, a normalization layer, and a deformable convolutional block, wherein the deformable convolutional block contains one convolutional layer and one deformable convolutional layer; each MobileViT_2 module includes a convolutional module, an unfolding module, a group normalization layer, a LinearAttnFFN module, and a folding module; each convolutional module includes a convolutional layer, a batch normalization layer, and a SiLU activation layer.
[0042] The input small molecule chemical structure image first passes through a 3×3 convolutional module, then through five cascaded MobileV2 blocks for image feature extraction. The second and fifth blocks downsample the input image while performing feature extraction. Then, it passes through a structure consisting of three MobileViT_2 blocks and two MobileV2 blocks with downsampling capabilities, which are arranged alternately, to achieve the fusion of local and global feature information. Finally, it passes through a 1×1 convolutional module and an average pooling layer.
[0043] In MobileViT_2block, the input image is first encoded in the local space through a 3x3 convolutional layer, and the number of channels in the network layer is adjusted through a 1x1 convolutional layer. Then, the input I1(H×W×d) is expanded into I2(P×N×d) by an unrolling module, where P = w×h is the size of the patch (w and h are the length and width of the patch, and h = w, which are 8, 4, and 2 in the three MobileViT_2blocks respectively), and N = HW / P is the number of patches (w and h need to be divisible by W and H respectively). A patch is viewed as... For a token, the token is flattened into a sequence. Then, L LinearAttnFFN modules replace local processing with global processing to extract global feature information, with the output being I3(P×N×d). After passing through a folding module, I4(H×W×d) with the same dimensions as I1 is obtained. The folding module can be seen as the inverse process of the unfolding module, that is, the image blocks are placed back into the original feature map. Finally, a 1×1 convolutional layer is used to adjust the number of channels back to the original size, and a 3×3 convolutional layer is used to fuse local and global features to achieve a global awareness operation.
[0044] In embodiments of the present invention, such as Figure 2 As shown, the first MobileV2 block module, the second MobileV2 block module, the third MobileV2 block module, the fourth MobileV2 block module, and the fifth MobileV2 block module are used to extract feature information from small molecule chemical structure images; among them, the second MobileV2 block module and the fifth MobileV2 block module are also used to downsample the small molecule chemical structure images.
[0045] In embodiments of the present invention, such as Figure 3 As shown, the first MobileViT_2block module, the second MobileViT_2block module, and the third MobileViT_2block module have the same structure, each including a first convolutional layer, a second convolutional layer, an unfolded layer, a first set of normalized layers, a LinearAttnFFN layer, a second set of normalized layers, a folded layer, a third convolutional layer, and a fourth convolutional layer connected in sequence.
[0046] In embodiments of the present invention, such as Figure 2As shown, the first MobileV2 block module, the second MobileV2 block module, the third MobileV2 block module, the fourth MobileV2 block module, the fifth MobileV2 block module, the sixth MobileV2 block module, and the seventh MobileV2 block module have the same structure, including a convolutional layer, a SiLU activation layer, a normalization layer, and a deformable convolutional block connected in sequence.
[0047] In this embodiment of the invention, the decoding part of the Conditional DETR network includes a first sublayer connection module, a second sublayer connection module, and a third sublayer connection module;
[0048] The original SMILES sequence is used as the input to the first and second sub-layer connection modules. The first and second sub-layer connection modules are used to normalize, learn features, and concatenate the original SMILES sequence. The third sub-layer connection module is used to decode the processing result of the second sub-layer connection module to generate SEFILES format characters, obtain the SELFIES sequence, and convert it into a new SMILES sequence through the selfies package.
[0049] like Figure 6 As shown, the decoding part of the Conditional DETR model serves as the decoding section of the entire encoder-decoder network. It consists of three stacked decoder layers, each composed of three sub-layer connection structures: the first sub-layer connection structure includes a layer normalization layer, a multi-head self-attention sub-layer, and a residual connection; the second sub-layer connection structure includes a layer normalization layer, a multi-head attention sub-layer, and a residual connection; the third sub-layer connection structure includes a layer normalization layer, a feedforward fully connected sub-layer, and a residual connection. In this part, all layer normalization layers are placed before the multi-head self-attention sub-layers.
[0050] In the input layer, the SMILES sequence is processed by vector mapping (encoding words into word vectors) and positional encoding, serving as the input for the first sub-layer. In the first sub-layer, the input sequence is normalized by a layer normalization layer, and then each token in the input sequence is learned and understood through a mask multi-head attention layer, establishing connections between tokens. The output value is then residually concatenated with the input of this sub-layer. The mask not only removes the influence of input padding, but also applies an upper triangular mask to the input to prevent the encoding part from seeing future information.
[0051] The second sub-layer takes the input SMILES as input. First, it normalizes the 2D coordinates of the reference point corresponding to the query and maps them to the same sinusoidal position encoding space as the spatial key to obtain a reference value. Then, it maps the offset information contained in the vector-mapped input of the first layer to a high-dimensional space through an FFN([linear+ReLU]+linear) to obtain the corresponding "offset". Then, it multiplies the reference value and the "offset" to obtain a tensor, which is concatenated with the tensor obtained by the batch normalization layer of the first layer. This concatenation is used as one of the inputs of the multi-head attention layer, and together with the output of the encoding part, it is used as the input of the multi-head attention layer to calculate the attention of all tokens in the sequence. Then, all values are concatenated to introduce more information in the context vector calculation. At the same time, the loss is reduced by reducing the dimensionality. The Mask removes the padding part of the image features and then performs a residual connection with the values before the second normalization.
[0052] Next, the output from the previous step is passed through a layer normalization layer and a feedforward layer, and then residually concatenated with the input of this sub-layer. Finally, it enters the output layer, where the FFN layer classifies the tokens based on the output and concatenates them to obtain the SELFIES sequence. This sequence then passes through a selfies decoder layer to decode each SELFIES format character into its corresponding SMILES format character, ultimately yielding the SMILES sequence.
[0053] In this embodiment of the invention, the first sub-layer connection module includes a first normalized layer, a first multi-head self-attention sub-layer, and a first residual layer; the second sub-layer connection module includes a second normalized layer, a second multi-head attention sub-layer, and a second residual layer; and the third sub-layer connection module includes a third normalized layer, a feedforward fully connected sub-layer, and a third residual layer.
[0054] This invention provides a small molecule chemical structure image recognition algorithm that performs excellently in terms of complex sequence prediction, model convergence speed, and learned weights, in order to solve the following problems:
[0055] 1) To address the issue of low prediction accuracy for complex sequences, this invention uses MobileViT_2 with the fully connected layers removed as the encoding part of the overall encoder-decoder network to extract feature information, especially local feature information, from the input chemical structure image. Furthermore, the depthwise separable convolution in the MobileV2 module of this model is replaced with deformable convolution (DCNv2). Compared with depthwise separable convolution, deformable convolution can have a receptive field and sampling position that are closer to the shape and size of the object, enabling the extraction of more accurate feature information. This allows subsequent processing to be performed with as much feature information as possible, resulting in better prediction performance for complex sequences in both objective and subjective evaluations.
[0056] 2) To address the issue of slow model convergence, this invention employs an encoder-decoder structure, which is more lightweight and effectively improves convergence speed. The decoding part of Conditional DETR is used as the decoding part of the overall encoder-decoder network to process the encoded results. The conditional fast DETR training cross-attention mechanism reduces the dependence on content embedding, simplifies training, and significantly improves the model's convergence speed. Simultaneously, the lightweight network MobileViT_2 is used as the encoding part, resulting in fewer model parameters. Furthermore, the internal structure of this network has been adjusted, placing the layer normalization in the multi-head attention mechanism of the encoding part before the attention mechanism. After these adjustments, the network is easier to optimize, has lower computational overhead, and effectively prevents gradient explosion or gradient vanishing.
[0057] 3) To address the issue of fluctuating learning rates and unstable learned weights, a learning rate warm-up approach can be adopted. This allows for a smaller learning rate in the first few batches or steps of training, enabling the model to gradually stabilize. Once the model is relatively stable, a pre-set learning rate can be selected for further training, resulting in faster convergence and more stable learned weights.
[0058] The effects of the present invention will be described below with reference to specific embodiments.
[0059] Comparing with reference 1, which uses Rajan, K., Zielesny, A., Steinbeck, C.: Decimer: towards deep learning for chemical image recognition. Journal of Cheminformatics 12 (2020). Figure 7The figure shown is a comparison of subjective evaluations. A comprehensive performance comparison was conducted between the method of this invention and the method in Reference 1, including: matching degree, two objective evaluation metrics for text generation (edit distance and BLEU-1), and three chemical fingerprint similarity metrics (MASSC fingerprint, topological fingerprint, and Morgan fingerprint). The data are shown in Table 1. Table 1 shows that the algorithm of this invention improves upon the other two algorithms in all metrics.
[0060] Table 1
[0061]
[0062] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.
Claims
1. A method for image recognition of small molecule chemical structures based on transform neural networks, characterized in that, Includes the following steps: S1: Acquire images of small molecule chemical structures and preprocess them; S2: Use the preprocessed small molecule chemical structure image as input to the MobileViT_2 network to extract the feature vector of the small molecule chemical structure image; In step S2, the MobileViT_2 network includes a first convolutional module, a first MobileV2 block module, a second MobileV2 block module, a third MobileV2 block module, a fourth MobileV2 block module, a fifth MobileV2 block module, a first MobileViT_2 block module, a sixth MobileV2 block module, a seventh MobileV2 block module, a third MobileViT_2 block module, a second convolutional module, and an average pooling module connected sequentially. The first, second, third, fourth, and fifth MobileV2 block modules are used to extract feature information from small molecule chemical structure images. The second and fifth MobileV2 block modules are also used to downsample the small molecule chemical structure images. The first, second, and third MobileViT_2 block modules are used to extract feature information from small molecule chemical structure images. The block modules have the same structure, each including a first convolutional layer, a second convolutional layer, an unfolded layer, a first set of normalized layers, a LinearAttnFFN layer, a second set of normalized layers, a folded layer, a third convolutional layer, and a fourth convolutional layer connected in sequence. S3: Obtain the original SMILES sequence, use it as a label, and together with the feature vector of the small molecule chemical structure image, use it as the input to the decoding part of the Conditional DETR network to obtain the SELFIES sequence. Then, use the selfies package to convert it into a new SMILES sequence as the recognition result to complete the recognition of the small molecule chemical structure image. The decoding part of the Conditional DETR network includes a first sublayer connection module, a second sublayer connection module, and a third sublayer connection module; The original SMILES sequence is used as the input to the first sub-layer connection module and the second sub-layer connection module; the first and second sub-layer connection modules are used to perform normalization, feature learning and concatenation on the original SMILES sequence; the third sub-layer connection module is used to decode the processing result of the second sub-layer connection module to generate SEFILES format characters, obtain the SELFIES sequence, and convert it into a new SMILES sequence through the selfies package. The first sub-layer connection module includes a first normalized layer, a first multi-head self-attention sub-layer, and a first residual layer; the second sub-layer connection module includes a second normalized layer, a second multi-head attention sub-layer, and a second residual layer; the third sub-layer connection module includes a third normalized layer, a feedforward fully connected sub-layer, and a third residual layer.
2. The method for small molecule chemical structure image recognition based on transform neural network according to claim 1, characterized in that, In step S1, the specific method for preprocessing the small molecule chemical structure image is as follows: convert the small molecule chemical structure image into an RGB image and then randomly rotate it.
3. The method for small molecule chemical structure image recognition based on transform neural network according to claim 1, characterized in that, The first, second, third, fourth, fifth, sixth, and seventh MobileV2 block modules have the same structure, including a convolutional layer, a SiLU activation layer, a normalization layer, and a deformable convolutional block connected in sequence.