A molecular image recognition method based on dual-branch multi-level feature decoding

Abstract

The present invention discloses a molecular image recognition method with dual-branch multi-level feature decoding. The molecular structure diagram to be identified is scaled into two images with different resolutions and respectively input into two convolutional neural networks with different complexities to obtain feature information of different spatial resolutions of the molecular structure diagram, and the InchI text corresponding to the molecular structure diagram is parsed through a Transformer decoder integrating a multi-level cross-attention mechanism. The molecular image recognition method with dual-branch multi-level feature Transformer decoding significantly and effectively improves the accuracy and robustness of molecular structure diagram recognition by efficiently and fully extracting features from large-size molecular images.

Description

Technical Field

[0001] The present invention relates to the field of molecular image recognition, and in particular to a molecular image recognition method based on dual-branch multi-level feature Transformer decoding. Background Art

[0002] The molecular skeleton structure diagram (Skeletal formula) is a structural symbol that has been used for hundreds of years and is also the most commonly used way for chemists to describe molecular structure. However, since most of the chemical information of molecular structure is presented in the form of images, the chemical meaning of the molecule is lost. Therefore, converting molecular structure diagrams into structured information data formats such as InchI that can be directly recognized by computers has attracted widespread attention in the scientific community.

[0003] As deep learning technology has achieved great success in the field of visual scene understanding in computer vision, some studies have attempted to use deep learning methods to identify molecular images.

[0004] At present, the most similar prior art to the present invention is that Ji Xiuyi, Li Jianhua and others proposed a molecular structure image recognition method based on deep learning in "Chemical Structure Image Recognition Based on Dual-Path Attention Mechanism". The deep neural network model consists of two parts: encoding and decoding. Specifically, ResNet is first used to extract features from the image and fine-tune the parameters to obtain feature vector information of the spatial feature map and the channel feature map. The two vectors are respectively subjected to the spatial attention mechanism and the channel attention mechanism to obtain two new feature map vector information and fuse them. The fused vector information is sent to the LSTM decoder for parsing.

[0005] In the image preprocessing stage, if the dataset images are unified into a small size and the large size images are reduced to small size images, the images are compressed and many molecular structure details are lost, and many features cannot be recognized. If the dataset images are unified into a large size and the currently common models are used for feature extraction, the required computing resources are very huge and not feasible for current computer hardware; if shallow neural networks are used to extract features from large size images, the image feature extraction capability of shallow neural networks is insufficient.

[0006] To solve the above problems, the present invention proposes a molecular image recognition method based on dual-branch multi-level feature Transformer decoding, which uses a dual-branch convolutional neural network to extract feature vector information of high-resolution images and low-resolution images, and adopts a Transformer decoder structure integrating a multi-level cross-attention mechanism to finally obtain a text representation of the molecular structure diagram. Summary of the invention

[0007] The present invention provides a molecular image recognition method with dual-branch multi-level feature Transformer decoding, thereby improving the problems of high hardware requirements and insufficient feature extraction that may exist in the current molecular image recognition methods for high-resolution molecular structure graph feature extraction, and improving the accuracy and robustness of model molecular image recognition.

[0008] To achieve the purpose of the present invention, a molecular image recognition method based on dual-branch multi-level feature Transformer decoding is provided, comprising the following steps:

[0009] The InchI text sequence is represented by discrete characters and cannot be directly input into the decoder model, so the InchI text sequence in the dataset needs to be converted into integer form first.

[0010] A dual-branch multi-level feature Transformer decoding model based on a neural network model is trained as a molecular structure diagram recognition method to improve the accuracy of molecular structure diagram recognition;

[0011] The obtained molecular structure diagram is input into the encoder to extract image features and obtain feature vector information of the image;

[0012] The vector features of the image are input into a decoder-Transformer (a deep learning model) decoder network model, and the decoder part extracts semantic information in combination with the feature information of the molecular structure image to obtain an InchI text sequence in the structured information data format of the molecular image;

[0013] The encoder is composed of a dual-branch convolutional neural network, and the decoder is a Transformer decoder structure that integrates a multi-level cross-attention mechanism;

[0014] Each molecular image in the molecular structure image dataset is scaled to two different resolutions and input into the model at the same time; for high-resolution molecular structure images, a convolutional neural network with fewer layers and a simpler model is used for feature extraction; for low-resolution images, a convolutional neural network with more layers and a more complex model is used for feature extraction. The above two-branch feature extraction models are connected in series to form the encoder feature extraction part;

[0015] Superimposing the extracted feature vectors with position codes having the same dimension as the extracted feature vectors to form new feature vectors;

[0016] A Transformer decoder model integrating a multi-level cross-attention mechanism is established to vectorize the InchI text corresponding to each molecular image and pass it into the model to obtain semantic information;

[0017] The vectorized InchI sequence is superimposed with the positional encoding, and the vector is passed to the Masked Multi-Head Attention layer. The mask is used to prevent the currently generated word from being dependent on the following words. Through the layer, the association between the currently generated word and the decoded word is captured, and a new vector is generated;

[0018] The feature vector is output from the masked multi-head attention layer and enters a multi-level cross-attention mechanism layer that interacts with the encoder. Here, two multi-head attention layers in series are used to interact with the encoder to deepen the connection between the image and the text.

[0019] Finally, in the decoder output part, the model obtains the probability distribution of the output through a linear transformation and a classifier layer, and then outputs the corresponding word with the highest probability as our predicted output through the dictionary. At the same time, each output of the decoder is used as the input of the decoder at the next moment;

[0020] When the output is completed, the InchI text sequence corresponding to the molecular structure image is obtained;

[0021] The performance of the molecular image recognition method of the present invention is evaluated using performance evaluation indicators.

[0022] The above-mentioned model based on dual-branch multi-level feature Transformer decoding includes a molecular structure image feature extraction part and an InchI sequence text generation part. Among them, for the molecular structure image feature extraction part, a dual-branch convolutional neural network is used to extract the feature vector information of high / low resolution images; for the InchI text sequence generation part, a Transformer decoder structure integrating a multi-level cross-attention mechanism is used to finally obtain the InchI sequence text of the molecular structure diagram. In this way, the embodiment of the present invention simply and automatically recognizes molecular structure images, improves the accuracy and robustness of large-size molecular structure image recognition, and thus improves the accuracy of molecular structure image recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a method for molecular structure graph recognition provided by an embodiment of the present invention.

[0024] Figure 2 The overall architecture diagram of the molecular structure diagram recognition provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0026] In order to simply and automatically identify molecular structure images and improve the accuracy of recognition, the present invention introduces a neural network model into the realization of molecular structure image recognition. With the rapid development of neural network models, neural network models can be applied to the recognition of molecular structure images. The neural network model of the embodiment of the present invention combines computer vision technology and text processing technology to generate InchI text sequences describing molecular images.

[0027] Specifically, the neural network model in the embodiment of the present invention adopts a model of dual-branch multi-level feature Transformer decoding based on the neural network model, and the model includes two main parts: molecular structure image feature extraction and InchI sequence text generation. Among them, for the molecular structure image feature extraction part, a dual-branch convolutional neural network is used to extract the feature vector information of high / low resolution images; for the InchI text sequence generation part, a Transformer decoder structure integrating a multi-level cross-attention mechanism is used to finally obtain the InchI sequence text of the molecular structure diagram.

[0028] Therefore, the embodiment of the present invention can simply and automatically recognize the molecular structure image and improve the accuracy of molecular structure image recognition.

[0029] Figure 1 A flow chart of a method for molecular structure diagram recognition provided by an embodiment of the present invention, wherein the steps are as follows:

[0030] Set up an encoder-decoder deep neural network as a molecular structure image recognition model, which includes an encoder part—image feature extraction and a decoder part—InchI text sequence automatic generation;

[0031] Here, the encoder part is composed of a dual-branch convolutional neural network, and the decoder part adopts the Transformer decoder structure that integrates a multi-level cross-attention mechanism;

[0032] The molecular structure image is input into the encoder part for feature extraction to obtain a feature vector, which is combined with the position code and sent to the decoder part. This embodiment adopts the sine-cosine encoding method, and the sine-cosine position encoding formula is as follows:

[0033]

[0034] The feature vector information of the image is sent to the decoder, and the decoder extracts semantic information based on the feature information of the image to obtain the InchI text corresponding to the molecular structure image.

[0035] In the examples of the present invention, the image feature extraction refers to a computer converting an image into a feature matrix or feature vector through a series of operations, usually represented by a vector of fixed length, which is the feature vector information of the image. Feature extraction usually uses convolutional neural networks, including MobileNet, ResNet, EfficientNet or AlexNet, etc.;

[0036] Figure 2 The overall architecture diagram of the molecular structure image recognition provided by the embodiment of the present invention, for high-resolution molecular structure images, a convolutional neural network MobileNet with fewer layers and a simpler model is used for feature extraction; for low-resolution images, a convolutional neural network EfficientNet B5 with more layers and a more complex model is used for feature extraction, including:

[0037] Convert the image into a high-resolution image and a low-resolution image for input;

[0038] The image is input into the high-resolution image feature extraction model and the low-resolution image feature extraction model;

[0039] The dual-branch feature extraction model extracts features from molecular images and obtains feature codes of fixed dimensions.

[0040] In an embodiment of the present invention, the decoder part is to predict the probability of the next InchI sequence text given the InchI text words that already exist in the sequence.

[0041] Since there are fewer characters in the InchI text sequence, but there are similarities between the characters (such as left and right brackets appear in pairs), in order to reflect the similarity between the characters, the model of the present invention uses character embedding to encode the characters. In addition to the characters of the InchI text, the start and end symbols ' <sos> ’，‘ <eos>'and completion symbol' <pad>', build a character dictionary.

[0042] The example of the present invention uses a Transformer decoder structure that integrates a dual-branch attention mechanism as a decoder. At each output time point, a new InchI text word is generated in the sequence, and then each generated InchI character is encoded using character embedding (word2Vec), and its fused position encoding is passed as input to the decoder for generating subsequent InchI characters.

[0043] The specific steps of the model decoder are:

[0044] Input fixed-dimensional feature encoding;

[0045] The feature code is fused with the position code vector to form a new feature code;

[0046] Input the new feature encoding vector into the decoder;

[0047] The decoder combines fixed-dimensional feature encoding to extract semantic information and outputs InchI sequence text.

[0048] It can be seen that the molecular structure diagram recognition model functions in the embodiment of the present invention include image feature extraction and InchI sequence text generation. The image features interact with the decoder part by inputting into the double-branch attention mechanism layer of the decoder. Assume that the InchI sequence text corresponding to the molecular structure diagram I is S, and the sequence length is n, that is, S = {S1, S2, ..., S n }, S1 in the embodiment of the present invention is <sos>,S n for <eos>. In an embodiment of the present invention, in the decoder part, the sequence text is vectorized and input into the multi-head attention mechanism to capture the association between the currently generated word and the decoded word, and generate a new vector; and the new vector is input into the dual-branch attention mechanism layer that interacts with the encoder part of the image feature model. Here, two multi-head attention layers in series are used to interact with the encoder to deepen the connection between the image and the text. Finally, in the decoder output part, the model obtains the output probability distribution through a linear transformation and a Softmax layer (a classifier layer), and then outputs the corresponding word with the highest probability as our predicted output through the dictionary. The architecture of the entire model is as follows: Figure 2 shown.

[0049] When training the recognition model of molecular structure images, inputting an image will output the InchI text sequence corresponding to the image, and the text sequence generation process is to generate a word at each moment, and the previously generated words will be used as input for the generation of subsequent words. In this way, a recognition model of molecular structure images is trained.

[0050] In the embodiment of the present invention, how to evaluate the quality of the recognition model of the molecular structure image obtained by training is an important issue. The embodiment of the present invention uses the Levenshtein Distance to evaluate the model. The Levenshtein Distance measures the minimum number of times a string A can be transformed into a string B through only three operations: insertion, deletion and substitution. The Levenshtein Distance has the following advantages: 1) It is easy to understand; 2) The matching degree obtained is high; 3) It is widely used in string matching.

[0051] When the edit distance is used to evaluate the trained InchI sequence text, the smaller the edit distance value is, that is, the more similar the obtained InchI sequence text is to the reference sequence text, the better the quality of the molecular structure image recognition system is.

[0052] It can be seen that the embodiment of the present invention constructs a molecular structure diagram recognition model that integrates computer vision technology and text processing technology. First, each molecular image is scaled to high / low resolution, and the image is subjected to high-resolution and low-resolution dual-branch feature extraction. A convolutional neural network is used. For high-resolution molecular structure images, a convolutional neural network MobileNet with fewer layers and a simpler model is used for feature extraction; for low-resolution images, a convolutional neural network EfficientNet B5 with more layers and a more complex model is used for feature extraction to obtain high-resolution feature vector information and low-resolution feature vector information of the feature map; secondly, text sequence processing is performed, and the word2ved trained model is integrated as the initialization of the word vector to extract semantic information together, and then a decoder integrating a multi-level cross-attention mechanism is used to fuse the dual-branch features extracted from the image with the output of the text sequence processing, and the fully connected layer is used for final prediction. In this way, higher accuracy and better fluency in molecular structure diagram recognition are achieved.

[0053] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.< / eos> < / sos> < / pad> < / eos> < / sos>

Claims

1. A molecular image recognition method with dual-branch multi-level feature decoding, characterized in that: The steps include: S10, scaling the molecular structure image to be identified into two images with different resolutions; S20, construct a dual-branch convolutional neural network model, where: a) For high-resolution images, a convolutional neural network with fewer layers and lower complexity is used; b) For low-resolution images, a convolutional neural network with more layers and higher complexity is used; S30, build a Transformer decoder model integrating multi-level cross attention mechanism; S40, combining the models in S20 and S30 to build a dual-branch multi-level feature Transformer decoding network architecture; S50, a molecular image recognition method using Transformer decoding with dual-branch multi-level features is used to obtain the InchI text corresponding to the molecular structure diagram.

2. A molecular image recognition method with dual-branch multi-level feature decoding as claimed in claim 1, characterized in that: The specific steps of step S20 to construct a dual-branch convolutional neural network model to extract feature information of two mixed images with different resolutions are: A dual-branch convolutional neural network model is used to extract feature information of images with different resolution mixtures. For high-resolution images, a convolutional neural network with fewer layers and lower complexity is used to extract molecular graph features. For low-resolution images, a convolutional neural network with more layers and higher complexity is used to extract molecular structure graph features.

3. A molecular image recognition method with dual-branch multi-level feature decoding as claimed in claim 1, characterized in that: The step S30 integrates the Transformer decoder model with the multi-level cross-attention mechanism, uses the Transformer decoder as the backbone network, and uses the multi-level cross-attention mechanism in the attention layer to replace the original cross-attention mechanism.

Images