Model training method based on sequence data and related product

Abstract

The invention provides a model training method based on sequence data and a related product. According to a specific implementation mode of the model training method based on the sequence data, Token (namely lexical element segmentation) is carried out based on a single molecular identifier, so that the number of lexical elements in a dictionary is still the number of types of the molecular identifiers, and a molecular identifier sequence coding model can be positioned to the single molecular identifier; therefore, downstream tasks of a single molecular identification level can be completed. Besides, at least one context vector is introduced into the vector representation of a single molecular identifier, and on the basis of the vector of the single molecular identifier, the context vectors are fused, the local perception ability of the model is enhanced, and the diversity of the input molecular identifier sequence coding model token is ensured, so that the problems that the dictionary capacity is too small and the model optimization is difficult are solved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of bioinformatics analysis, and in particular to a model training method based on sequence data and related products. Background Art

[0002] With the development of large model technology, the Transformer model has achieved great success in the field of text and its related multimodal fields. Inspired by this, researchers in the field of life sciences have also introduced the Transformer architecture to pre-train the encoding of molecular identifier sequences. In the pre-training process, it is usually necessary to establish a dictionary composed of word units based on the training data, and tokenize the molecular identifier sequence (or word unit segmentation) based on the dictionary, and then pre-train based on the tokenized data.

[0003] However, due to the small number of molecular identifiers involved in molecular identifier sequence data, the dictionary established based on molecular identifier sequence training data is also small. For example, the base identifier sequence corresponding to RNA consists of four letters: A, U, C, and G. If the dictionary is directly pre-trained with the four base identifiers A, U, C, and G as word units, a slight change in the vector representation data of any base identifier will cause a significant impact on other positions in the sequence, making model optimization very difficult.

[0004] To solve this problem, researchers have tried to tokenize based on k-mer segmentation. For example, using the 3-mer method, three consecutive base identifiers in the RNA molecule are mapped to a token. For example, AUC is mapped to a token, and then there will be 3 in the dictionary. 4 In this way, the number of words in the dictionary can be increased, that is, the number of tokens can be increased. However, when a single molecular identifier (e.g., a single base identifier) ​​needs to be located in the downstream task of the molecular identifier sequence encoding model (e.g., RNA structure prediction task), this approach maps the fragment of the molecular identifier sequence into a token and cannot locate a single molecular identifier, so the downstream task at the single molecular identifier level (e.g., base level) cannot be completed accordingly. Summary of the invention

[0005] The embodiments of the present disclosure provide a method, apparatus, electronic device, storage medium and computer program product for model training and sequence data encoding based on sequence data.

[0006] In a first aspect, an embodiment of the present disclosure provides a model training method based on sequence data, the method comprising:

[0007] Obtain sample molecular identification sequence;

[0008] Determine a vector and at least one context vector for each molecular identifier in the sample molecular identifier sequence;

[0009] Determining an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector;

[0010] Based on the input vector of each molecular identifier in the sample molecular identifier sequence, a preset sequence encoding model is pre-trained to obtain a molecular identifier sequence encoding pre-training model.

[0011] In some optional embodiments, determining the vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector includes:

[0012] Determining a vector of each molecular identifier in the sample molecular identifier sequence;

[0013] For each molecular identifier in the sample molecular identifier sequence, the following context vector extraction operation is performed at least once to obtain at least one context vector of the molecular identifier: feature extraction is performed on the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier, wherein the context vector sequence of the molecular identifier includes vectors of a first preset number of molecular identifiers adjacent to the molecular identifier above, the vector of the molecular identifier, and vectors of a second preset number of molecular identifiers adjacent to the molecular identifier below, arranged in sequence.

[0014] In some optional embodiments, the extracting features of the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier includes:

[0015] A convolution operation is performed on the context vector sequence of the molecular identifier to obtain the context vector of the molecular identifier.

[0016] In some optional embodiments, determining the input vector of the corresponding molecular identifier based on the vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector includes:

[0017] For each molecular identifier in the sample molecular identifier sequence, the sum of the vector of the molecular identifier and at least one context vector is determined as the input vector of the corresponding molecular identifier.

[0018] In some optional embodiments, the preset sequence encoding model includes a multi-head attention layer, a convolutional layer, and a feedforward neural network layer connected sequentially.

[0019] In some optional embodiments, the local neighborhood feature extraction layer is used to perform local neighborhood feature extraction on the attention weight map output by the multi-head attention layer to obtain a neighborhood attention weight map, and the feedforward neural network layer is used to process based on the neighborhood attention weight map to obtain features corresponding to each of the molecular identifiers.

[0020] In a second aspect, an embodiment of the present disclosure provides a sequence data encoding method, the method comprising:

[0021] Obtaining a molecular identification sequence to be encoded;

[0022] Determine a vector and at least one context vector for each molecular identifier in the sequence of molecular identifiers to be encoded;

[0023] Determining an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sequence of molecular identifiers to be encoded and at least one context vector;

[0024] The input vector of each molecular identifier in the molecular identifier sequence to be encoded is input into a molecular identifier sequence encoding pre-training model to obtain a molecular identifier feature sequence, wherein the molecular identifier sequence encoding pre-training model is pre-trained by the method described in any implementation method of the first aspect.

[0025] In a third aspect, an embodiment of the present disclosure provides a model training device based on sequence data, the device comprising:

[0026] A first acquisition module is configured to acquire a sample molecule identification sequence;

[0027] A first context vector determination module, configured to determine a vector and at least one context vector for each molecule identifier in the sample molecule identifier sequence;

[0028] A first input vector determination module is configured to determine an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector;

[0029] The pre-training module is configured to pre-train a preset sequence encoding model based on an input vector of each molecular identifier in the sample molecular identifier sequence to obtain a molecular identifier sequence encoding pre-training model.

[0030] In some optional implementations, the first context vector determination module is further configured to:

[0031] Determining a vector of each molecular identifier in the sample molecular identifier sequence;

[0032] For each molecular identifier in the sample molecular identifier sequence, the following context vector extraction operation is performed at least once to obtain at least one context vector of the molecular identifier: feature extraction is performed on the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier, wherein the context vector sequence of the molecular identifier includes vectors of a first preset number of molecular identifiers adjacent to the molecular identifier above, the vector of the molecular identifier, and vectors of a second preset number of molecular identifiers adjacent to the molecular identifier below, arranged in sequence.

[0033] In some optional embodiments, the extracting features of the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier includes:

[0034] A convolution operation is performed on the context vector sequence of the molecular identifier to obtain the context vector of the molecular identifier.

[0035] In some optional implementations, the first input vector determination module is further configured to:

[0036] For each molecular identifier in the sample molecular identifier sequence, the sum of the vector of the molecular identifier and at least one context vector is determined as the input vector of the corresponding molecular identifier.

[0037] In some optional embodiments, the preset sequence encoding model includes a sequentially connected multi-head attention layer, a local neighborhood feature extraction layer, and a feedforward neural network layer.

[0038] In some optional embodiments, the local neighborhood feature extraction layer is used to perform local neighborhood feature extraction on the attention weight map output by the multi-head attention layer to obtain a neighborhood attention weight map, and the feedforward neural network layer is used to process based on the neighborhood attention weight map to obtain features corresponding to each of the molecular identifiers.

[0039] In a fourth aspect, an embodiment of the present disclosure provides a sequence data encoding device, the device comprising:

[0040] A second acquisition module is configured to acquire a molecular identification sequence to be encoded;

[0041] A second context vector determination module is configured to determine a vector of each molecule identifier in the sequence of molecule identifiers to be encoded and at least one context vector;

[0042] A second input vector determination module is configured to determine an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sequence of molecular identifiers to be encoded and at least one context vector;

[0043] The encoding module is configured to input the input vector of each molecular identifier in the molecular identifier sequence to be encoded into a molecular identifier sequence encoding pre-training model to obtain a molecular identifier feature sequence, wherein the molecular identifier sequence encoding pre-training model is pre-trained by the method described in any implementation of the first aspect.

[0044] In some optional embodiments, the sequence data encoding device also includes: a decoding module, which is configured to input the molecular identification feature corresponding to each molecular identification in the molecular identification sequence to be encoded into a downstream task decoder to obtain the downstream task execution result corresponding to the molecular identification.

[0045] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect and / or the second aspect.

[0046] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner in the first aspect and / or the second aspect.

[0047] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method described in any implementation manner in the first aspect and / or the second aspect.

[0048] In order to solve the problem that the dictionary is too small or a single molecular identifier cannot be located in the current process of encoding a molecular identifier sequence, the model training method, device, electronic device, storage medium and computer program product based on sequence data provided by the embodiments of the present disclosure obtain a sample molecular identifier sequence, then determine the vector and at least one context vector of each molecular identifier in the sample molecular identifier sequence, and then determine the input vector of the corresponding molecular identifier based on the vector and at least one context vector of each molecular identifier in the sample molecular identifier sequence, and finally pre-train the preset sequence encoding model based on the input vector of each molecular identifier in the sample molecular identifier sequence to obtain a molecular identifier sequence encoding pre-training model. That is, by tokenizing (i.e., word unit segmentation) based on a single molecular identifier, the number of word units in the dictionary is still the number of types of molecular identifiers (for example, for RNA molecular sequences, the dictionary still includes 4 word units, i.e., four base identifiers A, U, C, and G), ensuring that the molecular identifier sequence encoding model can locate a single molecular identifier, and thus can complete downstream tasks at the level of a single molecular identifier. In addition, by introducing at least one context vector (or called N-Gram enhanced token representation) into the vector representation of a single molecular identifier, the context vector is fused on the basis of the single molecular identifier vector to ensure the diversity of the molecular identifier sequence encoding model tokens, thereby alleviating the problem of small dictionary capacity and difficulty in model optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0050] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0051] Figure 2A is a flowchart of an embodiment of a model training method based on sequence data according to the present disclosure;

[0052] Figure 2B is a decomposed flow chart of one embodiment of step 202 according to the present disclosure;

[0053] Figure 3 is a specific example of the model training method based on sequence data according to the present disclosure;

[0054] Figure 4 The left side is a schematic diagram of the secondary structure of an RNA sample, and the right side is a contact map of the secondary structure of the RNA sample;

[0055] Figure 5 is a schematic diagram of performing a convolution operation on an attention weight map according to the present disclosure;

[0056] Figure 6 is a flow chart of an embodiment of a sequence data encoding method according to the present disclosure;

[0057] Figure 7 is a structural schematic diagram of an embodiment of a model training device based on sequence data according to the present disclosure;

[0058] Figure 8 is a structural schematic diagram of an embodiment of a sequence data encoding device according to the present disclosure;

[0059] Fig. 9 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0061] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the sequence data-based model training and sequence data encoding methods, apparatuses, electronic devices, and storage media of the present disclosure can be applied.

[0063] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0064] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as model training applications based on sequence data, sequence data encoding applications, etc.

[0065] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices (e.g., keyboard, mouse, touch screen, microphone, camera, etc.) and information output devices (e.g., display screen, speaker, etc.), including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, used to provide model training services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0066] In some cases, the model training method based on sequence data provided by the present disclosure may be executed by the terminal devices 101, 102, 103, and accordingly, the model training apparatus based on sequence data may be provided in the terminal devices 101, 102, 103. In this case, the system architecture 100 may not include the server 105.

[0067] In some cases, the model training and sequence data encoding methods based on sequence data provided in the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining a sample molecular identification sequence" can be performed by the terminal devices 101, 102, 103, and the steps of "determining a vector and at least one context vector for each molecular identification in the sample molecular identification sequence" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the model training device based on sequence data can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0068] In some cases, the sequence data-based model training and sequence data encoding methods provided in the present disclosure can be executed by the server 105. Accordingly, the sequence data-based model training and sequence data encoding devices can also be set in the server 105. In this case, the system architecture 100 may also not include the terminal devices 101, 102, and 103.

[0069] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0070] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0071] Continue to refer Figure 2A , which shows a process 200 of an embodiment of a model training method based on sequence data according to the present disclosure, the model training method based on sequence data comprises the following steps:

[0072] Step 201, obtaining a sample molecular identification sequence.

[0073] Here, the sample molecular identification sequence can be composed of at least two molecular identifications arranged in order. The sample molecular identification sequence can be, for example, a base identification sequence of RNA, a base identification sequence of DNA, or an amino acid identification sequence of protein. Among them, the base identification is used to uniquely identify different bases. The amino acid identification is used to uniquely identify different amino acids. That is, the molecular identification can be a base identification or an amino acid identification.

[0074] Step 202: Determine a vector of each molecule identifier in the sample molecule identifier sequence and at least one context vector.

[0075] Here, for each molecular identifier in the sample molecular identifier sequence, a vector corresponding to the molecular identifier and at least one context vector of the molecular identifier may be determined.

[0076] The vector of the molecular identifier is used to represent the molecular identifier. Optionally, the vector of the molecular identifier can also be used to represent the position of the molecular identifier in the sample molecular identifier sequence.

[0077] The context vector of the molecular identifier is used to represent at least one preceding molecular identifier and at least one following molecular identifier that is adjacent to or not adjacent to the molecular identifier.

[0078] Here, various implementations may be used to determine the context vector of the molecule identifier.

[0079] For example, at least one context vector of the molecular identifier may include the vector of each of the third preset number (e.g., 1, 2, or 3) molecular identifiers adjacent to the molecular identifier above and the vector of each of the fourth preset number (e.g., 1, 2, or 3) molecular identifiers adjacent to the molecular identifier below. In this way, the vector of the molecular identifier and at least one context vector are the vectors of the consecutive (third preset number + 1 + fourth preset number) molecular identifiers in the sample molecular identifier sequence.

[0080] In some optional implementations, step 202 may include: Figure 2B The following steps 2021 and 2022 are shown:

[0081] Step 2021, determining the vector of each molecular identifier in the sample molecular identifier sequence.

[0082] Here, various implementations may be used to determine the vector of each molecule identifier in the sample molecule identifier sequence.

[0083] Optionally, step 2021 may be performed as follows: the sum of the self-embedding vector and the position embedding vector of each molecular identifier in the sample molecular identifier sequence is determined as the vector of the corresponding molecular identifier.

[0084] Here, the self-embedding vector of the molecular identifier can be pre-trained using algorithms such as Word2Vec and Glove, or can be pre-set as a random vector and then optimized and adjusted in subsequent steps.

[0085] The position embedding vector of the molecular identifier is used to characterize the position of the molecular identifier in the sample molecular identifier sequence. In practice, various position embedding vector representation methods can be used to embed the position of the molecular identifier in the sample molecular identifier sequence. For example, it can be preset as a random vector and trained and optimized in subsequent steps, or it can be calculated using a preset position calculation formula.

[0086] Here, the vector dimensions of the self-embedding vector and the position embedding vector of the molecule identifier may be the same.

[0087] Step 2022: For each molecular identifier in the sample molecular identifier sequence, perform a context vector extraction operation at least once to obtain at least one context vector for the molecular identifier.

[0088] Here, the context vector extraction operation may be performed as follows: feature extraction is performed on the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier. At least one context vector of the molecular identifier may be obtained by performing the context vector extraction operation at least once.

[0089] Among them, the context vector sequence of the molecular identifier includes the vector of each molecular identifier in the first preset number of molecular identifiers adjacent to the molecular identifier above, the vector of the molecular identifier, and the vector of each molecular identifier in the second preset number of molecular identifiers adjacent to the molecular identifier below, which are arranged in sequence. That is, feature extraction is performed by performing feature extraction on a context vector sequence composed of vectors of continuous (first preset number + 1 + second preset number) molecular identifiers to obtain a context vector of the molecular identifier. In this way, a context vector of the molecular identifier represents the molecular identifier itself and its first preset number of molecular identifiers above and the second preset number of molecular identifiers below.

[0090] Optionally, here, the first preset number and the second preset number may be the same value. For example, the first preset number and the second preset number may both be 1, or both be 2, or both be 3.

[0091] It can be understood that when the first preset number and the second preset number are both 1, the context vector sequence of the molecular identifier is composed of three consecutive molecular identifiers centered on the molecular identifier arranged in sequence. The context vector obtained by feature extraction of the context vector sequence of the molecular identifier can be understood as a 3-Gram vector.

[0092] When the first preset number and the second preset number are both 2, the context vector sequence of the molecular identifier is composed of vectors of five consecutive molecular identifiers centered on the molecular identifier arranged in sequence. The context vector obtained by feature extraction of the context vector sequence of the molecular identifier can be understood as a 5-Gram vector.

[0093] When the first preset number and the second preset number are both 3, the context vector sequence of the molecular identifier is composed of seven consecutive molecular identifier vectors arranged in sequence with the molecular identifier as the center. The context vector obtained by feature extraction of the context vector sequence of the molecular identifier can be understood as a 7-Gram vector.

[0094] It should be noted that, here, during at least one execution of the above context vector extraction operation, at least one of the first preset number and the second preset number of two different context vector extraction operations is different, for example, the first preset number is different, or the second preset number is different, or both the first preset number and the second preset number are different. Each execution obtains a context vector of the molecular identifier, and the execution is performed several times to obtain several upper and lower vectors of the molecular identifier.

[0095] As an example, here, three context vector extraction operations can be performed to obtain three context vectors of the molecular identifier. Among them, including context vector extraction operations in which the first preset number and the second preset number are both 1, context vector extraction operations in which the first preset number and the second preset number are both 2, and context vector extraction operations in which the first preset number and the second preset number are both 3. Among them, a 3-Gram vector can be obtained through the context vector extraction operation in which the first preset number and the second preset number are both 1, a 5-Gram vector can be obtained through the context vector extraction operation in which the first preset number and the second preset number are both 2, and a 7-Gram vector can be obtained through the context vector extraction operation in which the first preset number and the second preset number are both 3. Thus, three context vectors of the molecular identifier can be obtained, namely: a 3-Gram vector, a 5-Gram vector and a 7-Gram vector.

[0096] Here, various time series feature extraction methods can be used to extract features from the context vector sequence of the molecular identification to obtain the context vector of the molecular identification, and the dimension of the context vector of the molecule is the same as the dimension of the vector of the molecular identification and the context vector. As an example, a recurrent neural network (RNN) can be used to extract features from the context vector sequence of the molecular identification.

[0097] In some optional implementations, feature extraction is performed on the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier, which may be performed as follows:

[0098] A convolution operation is performed on the context vector sequence of the molecular identifier to obtain the context vector of the molecular identifier.

[0099] As an example, assume that the vector of each molecular identifier is a D-dimensional vector, the first preset number is N1, and the second preset number is N2. When the context vector sequence of the molecular identifier is composed of the vectors of the above consecutive (first preset number N1+1+second preset number N2) molecular identifiers, the context vector sequence of the molecular identifier can be expressed as a matrix of D*(N1+1+N2). Then, the convolution operation on the above context vector sequence can be: use D different convolution kernels of size 1*(N1+1+N2), and convolve each of the D rows in the above matrix M with a corresponding convolution kernel of size 1*(N1+1+N2). A scalar corresponding to the row can be obtained, and then a D-dimensional vector of the D-dimensional molecular identifier can be obtained, and then the above D-dimensional vector is nonlinearly transformed and mapped to obtain a D-dimensional context vector.

[0100] Step 203: Determine an input vector of a corresponding molecular identifier based on the vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector.

[0101] Here, various implementations may be used to determine the input vector of the corresponding molecular identifier based on the vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector.

[0102] For example, for each molecular identifier in the sample molecular identifier sequence, a mean vector of the vector of the molecular identifier and at least one context vector may be determined as the input vector of the corresponding molecular identifier.

[0103] Optionally, for each molecular identifier in the sample molecular identifier sequence, the sum of the vector of the molecular identifier and at least one context vector can be determined as the input vector of the corresponding molecular identifier. In this way, the input vector of the molecular identifier represents both the molecular identifier and the context molecular identifier of the molecular identifier.

[0104] Step 204 , based on the input vector of each molecular identifier in the sample molecular identifier sequence, pre-train the preset sequence encoding model to obtain a molecular identifier sequence encoding pre-training model.

[0105] Here, various models suitable for encoding a vector sequence can be used as the preset sequence encoding model. For example, the preset sequence encoding model can be an encoder in a Transformer, and so on.

[0106] By inputting the input vector of each molecular identifier in the sample molecular identifier sequence into a preset sequence encoding model, feature encoding can be performed on the input vector of the molecular identifier, and a feature vector sequence corresponding to the sample molecular identifier sequence can be output.

[0107] During the pre-training process, the input vector of each molecular identifier in the sample molecular identifier sequence can be input into a preset sequence encoding model, and various self-supervised training methods can be used to perform self-supervised training on the preset sequence encoding model to obtain a molecular identifier sequence encoding pre-training model.

[0108] As an example, the above self-supervised training method may include but is not limited to: mask pre-training and contrastive learning.

[0109] It should be noted that the above steps 201 to 204 only illustrate the process of pre-training a preset sequence encoding model based on a sample molecule identification sequence. In practice, multiple sample molecule identification sequences can be used or the preset sequence encoding model can be pre-trained in multiple batches. The present disclosure does not make specific limitations or elaborate on this.

[0110] The molecular identifier sequence encoding pre-training model trained by the method of step 201 to step 204 can be used to encode the molecular identifier sequence to obtain a feature sequence, and because the molecular identifier sequence encoding pre-training model uses a single molecular identifier as a word unit to perform word unit segmentation and vectorization during the training process, a single molecule level task is implemented in subsequent downstream tasks. In addition, the input vector of each molecular identifier also includes a context vector, which can improve the word unit diversity input to the preset sequence encoding model and alleviate the problem of model optimization with an overly small dictionary.

[0111] Specifically, please refer to Figure 3 , Figure 3 A specific example of a model training method based on sequence data according to the present disclosure is shown. Figure 3 As shown, the sample molecular identification sequence is the RNA molecular identification sequence "AUCMAGCM", where M indicates that the base at that position is masked. Each molecular identification in the above RNA molecular identification sequence corresponds to a corresponding D (eg, 128) dimensional vector.

[0112] For each molecular identifier in the above RNA molecular identifier sequence, three context vector extraction operations may be performed to obtain three context vectors of the molecular identifier.

[0113] The following uses the second "A" in "AUCMAGCM" as an example to illustrate the process of obtaining the input vector corresponding to each molecular identifier:

[0114] The first context vector extraction operation: Use D convolution kernels of size 1*3 to convolve the vectors of three consecutive molecular identifiers (i.e., "MAG") centered on the molecular identifier, and obtain a 3-Gram vector V1 after passing through a nonlinear function. The dimension of V1 is D.

[0115] The second context vector extraction operation: Use D convolution kernels of size 1*5 to convolve the vectors of five consecutive molecular identifiers (i.e., "CMAGC") centered on the molecular identifier, and obtain a 5-Gram vector V2 after passing through a nonlinear function. The dimension of V2 is D.

[0116] The third context vector extraction operation: Use D convolution kernels of size 1*7 to convolve the vectors of seven consecutive molecular identifiers (i.e., "UCMAGCM") centered on the molecular identifier, and obtain a 7-Gram vector V3 after passing through a nonlinear function. The dimension of V3 is D.

[0117] Add the vector of the second "A" in "AUCMAGCM" to the above-mentioned 3-Gram vector F1, 5-Gram vector F2 and 7-Gram vector F3 to obtain the input vector V of the second "A" in "AUCMAGCM".

[0118] By performing the above three context vector extraction operations on each RNA molecule identifier in "AUCMAGCM", the 3-Gram vector V1, 5-Gram vector V2 and 7-Gram vector V3 of each RNA molecule identifier are obtained respectively, and the vector of each RNA molecule identifier, 3-Gram vector V1, 5-Gram vector V2 and 7-Gram vector V3 are added to obtain the input vector V of the corresponding RNA molecule identifier. The input vector V of each RNA molecule identifier in "AUCMAGCM" is synchronously input into the preset sequence encoding model, and the feature F of each RNA molecule identifier in "AUCMAGCM" can be output.

[0119] There is folding between RNA and protein molecules, and after folding, a certain secondary / tertiary structure is formed. For example, in the secondary / tertiary structure of RNA, there are interactions between molecules (base pairing). These interaction units in RNA molecules usually have certain aggregation characteristics (group wise), that is, one sequence interacts with another sequence, and these interaction units usually do not exist between bases with close sequence distances.

[0120] Please refer to Figure 4 , Figure 4 The left side is a schematic diagram of the secondary structure of an RNA sample, and the right side is a secondary structure contact map of the RNA sample. Among them, the horizontal and vertical coordinates in the secondary structure contact map correspond to different molecular positions in the RNA molecular sequence, and the points in the secondary structure contact map are used to indicate that there is an interaction between the RNA molecule at the horizontal coordinate of the point and the RNA molecule at the vertical coordinate. The line segment connected by continuous points in the secondary structure contact map indicates that there is an interaction between one molecular sequence and another molecular sequence. The points of the white diagonal line in the secondary structure contact map correspond to the same molecular position. The points closer to the white diagonal line in the secondary structure contact map indicate that there is an interaction between two RNA molecules that are closer, and the points farther from the diagonal line indicate that there is an interaction between two bases that are farther away. It can be seen from the above secondary structure contact map that there are multiple line segments farther from the diagonal line, that is, it indicates that there is an interaction between two molecular sequences that are farther away.

[0121] Based on the above findings, reasonable modeling of this interaction with aggregation characteristics will help improve the effect of molecular marker sequence encoding pre-training models in RNA structure prediction. At the same time, in molecules, the sequence determines the structure, and the structure further determines the function. A better encoding structure helps improve all downstream tasks related to RNA property prediction. Most existing models for encoding text use a multi-head attention mechanism to learn the influence weight between any two positions in a character sequence, but there is no interaction pattern in the text similar to the above RNA molecular sequence. Most of the text is similar to Figure 4 The points around the diagonal line in the contact graph, that is, the characters that are closer have a higher influence weight, while the characters that are farther away have a lower influence weight. Therefore, introducing the prior information of the interaction between RNA bases into the multi-head attention mechanism can better model the interaction of the unique aggregation characteristics in RNA molecules.

[0122] Therefore, in order to characterize the interactions with aggregation characteristics between RNA molecules that are far away from the diagonal line in the above-mentioned contact diagram, in some optional embodiments, the above-mentioned preset sequence encoding model may include a sequentially connected multi-head attention layer, a local neighborhood feature extraction layer and a feedforward neural network layer.

[0123] Here, the multi-head attention layer is used to extract the attention weight map for the input vector of each molecular identifier in the input sample molecular identifier sequence. Among them, the attention weight map is used to characterize the attention weight of one molecular identifier input vector to another molecular identifier input vector in the two molecular identifier input vectors. According to the above description, there are multiple line segments far from the diagonal line in the secondary structure contact map of the RNA molecule, that is, it indicates that there is an interaction between two molecular sequences far away, but the attention weight map input by the multi-head attention layer cannot characterize this situation. Therefore, a neighborhood feature extraction layer is added on the basis of the multi-head attention layer. The neighborhood feature extraction layer is used to extract local neighborhood features from the attention weight map output by the multi-head attention layer to obtain a neighborhood attention weight map. The feedforward neural network layer is used to process based on the neighborhood attention weight map to obtain features corresponding to each molecular identifier. For example, the neighborhood attention weight map can be first processed by some processing (for example, normalization, residual connection processing, etc.), and then the processing result is input to the feedforward neural network layer to complete the final encoding work to output the model-encoded features of each molecular identifier.

[0124] Here, since the neighborhood attention weight map can characterize the interaction between the two molecular sequences of the above-mentioned RNA molecule, the feature expression dimension of the model is improved, and ultimately the feature encoding ability of the model can be improved to help improve the effect of downstream tasks.

[0125] Here, various neighborhood feature extraction methods may be used to extract neighborhood features from the attention weight map, examples of which may include but are not limited to: convolutional neural networks.

[0126] Alternatively, it can be understood that the preset sequence encoding model is based on the transformer encoder architecture, and the attention weight map (attention map) output by the multi-head attention layer is convolved to obtain the neighborhood attention weight map, and the neighborhood attention weight map is used to replace the attention weight map output by the multi-head attention layer in the original transformer encoder, and other operations of the transformer encoder are continued. For example, the subsequent Softmax normalization and the dot product of the value weight matrix can be continued to obtain the value matrix, and then the value matrix can be input into the normalization layer (Add&Norm), the feedforward neural network layer (Feed Forward) and the normalization layer (Add&Norm) in sequence.

[0127] Please refer to Figure 5 ,like Figure 5 As shown, the attention weight map (attention map) output by the original transformer's multi-head attention layer is an L*L matrix QK T , where L is the longest length of the molecular identifier sequence that the preset sequence encoding model can accept, Q is the query matrix, and K is the key matrix. T The element QK in the i-th row and j-th column T i,j is used to characterize the attention weight of the i-th molecular identifier to the j-th molecular identifier in the input molecular identifier sequence.

[0128] In order to indicate that there is an interaction between the i-th molecule and the j-th molecule in the contact graph, is there a point with coordinates (i, j) in the contact graph? Figure 4 The line segments formed by the points shown on the right side of the figure, that is, what points are around the coordinate point (i, j), and whether some of them can be connected to form a line segment, that is, whether there are other two interacting points (or two bases, two RNA molecules) around the coordinate point (i, j). By using the convolution operation, the original attention weight map is pattern enhanced, and the interactions with aggregation characteristics between bases are learned in the local range of the convolution kernel.

[0129] It should be noted that various sizes of convolution kernels and different numbers of convolution kernels can be used for combined convolution, as long as the neighborhood attention weight map after convolution is the same size as the original attention weight map. Optionally, the neighborhood attention weight map is made the same size as the original attention weight map by using the Padding method. For example, a 3*3 convolution kernel, a 5*5 convolution kernel, and a 7*7 convolution kernel, or a combination of the above convolution kernels, can be used to perform a convolution operation on the original attention weight map.

[0130] Since the molecular sequence determines the molecular structure, and the molecular structure determines the molecular function. Therefore, if the molecular identifier sequence encoding model can encode the molecular structure, that is, solve the problem of molecular structure, then the molecular identifier sequence encoding model can also solve downstream functional tasks, such as attribute prediction tasks. Therefore, here, by performing a convolution operation on the original attention weight map in transformer pre-training to extract two-dimensional features, that is, extracting two-dimensional structural features, it is equivalent to extracting which positions in the contact map have fragment-level intermolecular correlation effects (i.e., pairing, base pairing), and then the obtained neighborhood attention weight map strengthens the fragment-level intermolecular interaction pattern, which makes transformer pre-training (e.g., mask restoration) easier.

[0131] The model training method based on sequence data provided by the above-mentioned embodiment of the present disclosure obtains the sample molecular identification sequence, then determines the vector and at least one context vector of each molecular identification in the sample molecular identification sequence, and then determines the input vector of the corresponding molecular identification based on the vector and at least one context vector of each molecular identification in the sample molecular identification sequence, and finally pre-trains the preset sequence encoding model based on the input vector of each molecular identification in the sample molecular identification sequence to obtain a molecular identification sequence encoding pre-training model. That is, by tokenizing (i.e., word unit segmentation) based on a single molecular identification, the number of word units in the dictionary is still the number of types of molecular identification (for example, for RNA molecular sequences, the dictionary still includes 4 word units, i.e., four base identifications of A, U, C, and G), ensuring that the molecular identification sequence encoding model can be located to a single molecular identification, and then the downstream tasks at the level of a single molecular identification can be completed. In addition, by introducing at least one context vector (or N-Gram enhanced token representation) in the vector representation of a single molecular identification, the context vector is integrated on the basis of a single molecular identification vector to ensure the diversity of the molecular identification sequence encoding model token, so as to alleviate the problem of too small dictionary capacity and difficult model optimization.

[0132] Embodiment 1:

[0133] In the specific experiment, the applicant referred to the experimental settings recorded in the paper entitled "BEACON: Benchmark for Comprehensive RNA Tasks and Language Models", used the transformer model to predict RNA structure, and in the process of pre-training the transformer, used Figure 3 The input vector generation method shown generates the input vector of each molecular identifier for pre-training and subsequent RNA structure prediction. Please refer to Table 1 for specific experimental results:

[0134] Table 1

[0135]

[0136] The third row in Table 1 shows the experimental results of the public experiment. It can be seen from Table 1 that after using the context vector of each RNA molecule identifier, the RNA structure prediction task is improved compared with the SOTA and BEACON in the paper.

[0137] Reference below Figure 6 , which shows a process 600 of an embodiment of a sequence data encoding method according to the present disclosure. The sequence data encoding method comprises the following steps:

[0138] Step 601, obtaining a molecular identification sequence to be encoded.

[0139] Here, the molecular identification sequence to be encoded can be composed of various molecular identification sequences arranged in order. As an example, the molecular identification sequence to be encoded can be a base identification sequence of RNA, a base identification sequence of DNA, or an amino acid identification sequence of protein.

[0140] Step 602: Determine a vector of each molecule identifier in the sequence of molecule identifiers to be encoded and at least one context vector.

[0141] Here, you can use Figure 2A The same or similar method described in step 202 in the illustrated embodiment determines the vector of each molecule identifier in the sequence of molecule identifiers to be encoded and at least one context vector, which will not be described in detail here.

[0142] Step 603: Determine an input vector of a corresponding molecular identifier based on the vector of each molecular identifier in the sequence of molecular identifiers to be encoded and at least one context vector.

[0143] Here, you can use Figure 2A The same or similar method as described in step 203 in the illustrated embodiment is used to determine the input vector of each molecular identifier, which will not be described in detail herein.

[0144] Step 604: input the input vector of each molecular identifier in the molecular identifier sequence to be encoded into the molecular identifier sequence encoding pre-training model to obtain a molecular identifier feature sequence.

[0145] Here, the molecular identification features in the molecular identification feature sequence correspond one-to-one to the molecular identifications in the molecular identification sequence to be encoded, that is, each molecular identification in the molecular identification sequence to be encoded has a unique corresponding molecular identification feature in the molecular identification feature sequence.

[0146] Here, the molecular identifier sequence encoding pre-training model is Figure 2A The model training method based on sequence data recorded in the illustrated embodiment and its optional implementation methods is pre-trained. Since the molecular identifier sequence encoding pre-training model performs tokenization based on a single molecular identifier during the pre-training process, and at least one context vector (or N-Gram enhanced token representation) is introduced into the vector representation of a single molecular identifier, the molecular identifier feature sequence obtained by the above molecular identifier sequence encoding pre-training model can locate the features of each molecular identifier, and then the downstream tasks at the level of a single molecular identifier can be completed subsequently.

[0147] In some optional implementations, the above sequence data encoding method may further include the following steps 605:

[0148] Step 605: for each molecular identifier in the molecular identifier sequence to be encoded, the molecular identifier feature corresponding to the molecular identifier is input into the downstream task decoder to obtain the downstream task execution result corresponding to the molecular identifier.

[0149] Here, the downstream task decoder can be various molecular-level downstream tasks, such as molecular property prediction tasks, molecular structure prediction tasks, etc.

[0150] By executing step 605 , downstream tasks at the molecular level can be achieved.

[0151] The sequence data encoding method provided by the above-mentioned embodiments of the present disclosure forms an input vector based on the vector and context vector of each molecular identifier in the molecular identifier sequence to be encoded, and the input vector is input into a molecular identifier sequence encoding pre-trained model that is also based on a single molecular identifier for tokenization. At the same time, at least one context vector (or called N-Gram enhanced token representation) is introduced into the vector representation of a single molecular identifier, so that the obtained molecular identifier feature sequence can locate the features of each molecular identifier, and subsequently the downstream tasks at the single molecular identifier level can be completed.

[0152] Further references Figure 7As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a model training device based on sequence data, and the device embodiment is similar to Figure 2A Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0153] like Figure 7 As shown, the model training device 700 based on sequence data of this embodiment includes: a first acquisition module 701, a first context vector determination module 702, a first input vector determination module 703 and a pre-training module 704. Among them, the first acquisition module 701 is configured to acquire a sample molecular identification sequence; the first context vector determination module 702 is configured to determine the vector of each molecular identification in the sample molecular identification sequence and at least one context vector; the first input vector determination module 703 is configured to determine the input vector of the corresponding molecular identification based on the vector of each molecular identification in the sample molecular identification sequence and at least one context vector; the pre-training module 704 is configured to pre-train the preset sequence encoding model based on the input vector of each molecular identification in the sample molecular identification sequence to obtain a molecular identification sequence encoding pre-training model.

[0154] In this embodiment, the specific processing of the first acquisition module 701, the first context vector determination module 702, the first input vector determination module 703 and the pre-training module 704 of the model training device based on sequence data and the technical effects brought by them can be referred to respectively. Figure 2A The relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment are not repeated here.

[0155] In some optional implementations, the first context vector determining module 702 may be further configured to:

[0156] Determining a vector of each molecular identifier in the sample molecular identifier sequence;

[0157] For each molecular identifier in the sample molecular identifier sequence, the following context vector extraction operation is performed at least once to obtain at least one context vector of the molecular identifier: feature extraction is performed on the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier, wherein the context vector sequence of the molecular identifier includes vectors of a first preset number of molecular identifiers adjacent to the molecular identifier above, the vector of the molecular identifier, and vectors of a second preset number of molecular identifiers adjacent to the molecular identifier below, arranged in sequence.

[0158] In some optional implementations, the extracting features of the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier may include:

[0159] A convolution operation is performed on the context vector sequence of the molecular identifier to obtain the context vector of the molecular identifier.

[0160] In some optional implementations, the first input vector determination module 703 may be further configured to:

[0161] For each molecular identifier in the sample molecular identifier sequence, the sum of the vector of the molecular identifier and at least one context vector is determined as the input vector of the corresponding molecular identifier.

[0162] In some optional embodiments, the preset sequence encoding model may include a multi-head attention layer, a convolutional layer, and a feedforward neural network layer connected sequentially.

[0163] In some optional embodiments, the local neighborhood feature extraction layer is used to perform local neighborhood feature extraction on the attention weight map output by the multi-head attention layer to obtain a neighborhood attention weight map, and the feedforward neural network layer is used to process based on the neighborhood attention weight map to obtain features corresponding to each of the molecular identifiers.

[0164] It should be noted that the implementation details and technical effects of each module in the molecular identification sequence encoding model device provided in the embodiment of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0165] Reference below Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a sequence data encoding device. Figure 6 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0166] like Figure 8As shown, the sequence data encoding device 800 of this embodiment includes: a second acquisition module 801, a second context vector determination module 802, a second input vector determination module 803 and an encoding module 804. The second acquisition module 801 is configured to acquire a sequence of molecular identifiers to be encoded; the second context vector determination module 802 is configured to determine the vector of each molecular identifier in the sequence of molecular identifiers to be encoded and at least one context vector; the second input vector determination module 803 is configured to determine the input vector of the corresponding molecular identifier based on the vector of each molecular identifier in the sequence of molecular identifiers to be encoded and at least one context vector; and the encoding module 804 is configured to input the input vector of each molecular identifier in the sequence of molecular identifiers to be encoded into a molecular identifier sequence encoding pre-training model to obtain a molecular identifier feature sequence, wherein the molecular identifier sequence encoding pre-training model is pre-trained by the method described in any implementation of the first aspect.

[0167] In this embodiment, the specific processing of the second acquisition module 801, the second context vector determination module 802, the second input vector determination module 803 and the encoding module 804 of the sequence data encoding device 800 and the technical effects thereof can be referred to respectively. Figure 6 The relevant descriptions of step 601, step 602, step 603 and step 604 in the corresponding embodiment are not repeated here.

[0168] In some optional embodiments, the sequence data encoding device 800 may further include: a decoding module 805, configured to input the molecular identification feature corresponding to each molecular identification in the molecular identification sequence to be encoded into a downstream task decoder to obtain the downstream task execution result corresponding to the molecular identification.

[0169] It should be noted that the implementation details and technical effects of each module in the molecular identification sequence encoding model device provided in the embodiment of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0170] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 suitable for implementing the electronic device of the present disclosure. Fig. 9 The computer system 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0171] like Fig. 9As shown, the computer system 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the computer system 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0172] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication devices 909 may allow the computer system 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The computer system 900 of an electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0173] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0174] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0175] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0176] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2A The embodiment and optional implementations thereof show the model training method based on sequence data and / or Figure 6 The illustrated embodiment and its optional implementations illustrate a method for encoding sequence data.

[0177] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Python, Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0178] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0179] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring a sample molecular identification sequence".

[0180] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A model training method based on sequence data, comprising: Obtain sample molecular identification sequence; Determine a vector and at least one context vector for each molecular identifier in the sample molecular identifier sequence; Determining an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector; Based on the input vector of each molecular identifier in the sample molecular identifier sequence, a preset sequence encoding model is pre-trained to obtain a molecular identifier sequence encoding pre-training model.

2. The method according to claim 1, wherein: The determining of the vector of each molecule identifier in the sample molecule identifier sequence and at least one context vector comprises: Determining a vector of each molecular identifier in the sample molecular identifier sequence; For each molecular identifier in the sample molecular identifier sequence, the following context vector extraction operation is performed at least once to obtain at least one context vector of the molecular identifier: feature extraction is performed on the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier, wherein the context vector sequence of the molecular identifier includes the vector of each molecular identifier in a first preset number of molecular identifiers adjacent to the molecular identifier above, the vector of the molecular identifier, and the vector of each molecular identifier in a second preset number of molecular identifiers adjacent to the molecular identifier below, arranged in sequence.

3. The method according to claim 2, wherein: The step of extracting features from the context vector sequence of the molecular identifier to obtain a context vector of the molecular identifier includes: A convolution operation is performed on the context vector sequence of the molecular identifier to obtain the context vector of the molecular identifier.

4. The method according to claim 1, wherein: The step of determining the input vector of the corresponding molecular identifier based on the vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector comprises: For each molecular identifier in the sample molecular identifier sequence, the sum of the vector of the molecular identifier and at least one context vector is determined as the input vector of the corresponding molecular identifier.

5. The method according to claim 1, wherein: The preset sequence encoding model includes a sequentially connected multi-head attention layer, a local neighborhood feature extraction layer and a feedforward neural network layer.

6. The method according to claim 5, wherein: The local neighborhood feature extraction layer is used to perform local neighborhood feature extraction on the attention weight map output by the multi-head attention layer to obtain a neighborhood attention weight map, and the feedforward neural network layer is used to process based on the neighborhood attention weight map to obtain features corresponding to each of the molecular identifiers.

7. A sequence data encoding method, comprising: Obtaining a molecular identification sequence to be encoded; Determine a vector and at least one context vector for each molecular identifier in the sequence of molecular identifiers to be encoded; Determining an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sequence of molecular identifiers to be encoded and at least one context vector; The input vector of each molecular identifier in the molecular identifier sequence to be encoded is input into a molecular identifier sequence encoding pre-training model to obtain a molecular identifier feature sequence, wherein the molecular identifier sequence encoding pre-training model is pre-trained by the method described in any one of claims 1 to 6.

8. A model training device based on sequence data, comprising: A first acquisition module is configured to acquire a sample molecule identification sequence; A first context vector determination module, configured to determine a vector and at least one context vector for each molecule identifier in the sample molecule identifier sequence; A first input vector determination module is configured to determine an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sample molecular identifier sequence and at least one context vector; The pre-training module is configured to pre-train a preset sequence encoding model based on an input vector of each molecular identifier in the sample molecular identifier sequence to obtain a molecular identifier sequence encoding pre-training model.

9. A sequence data encoding device, comprising: A second acquisition module is configured to acquire a molecular identification sequence to be encoded; A second context vector determination module is configured to determine a vector of each molecule identifier in the sequence of molecule identifiers to be encoded and at least one context vector; A second input vector determination module is configured to determine an input vector of a corresponding molecular identifier based on a vector of each molecular identifier in the sequence of molecular identifiers to be encoded and at least one context vector; The encoding module is configured to input the input vector of each molecular identifier in the molecular identifier sequence to be encoded into a molecular identifier sequence encoding pre-training model to obtain a molecular identifier feature sequence, wherein the molecular identifier sequence encoding pre-training model is pre-trained by the method described in any one of claims 1 to 6.

10. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6 and / or the method according to claim 7.

11. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 6 and / or the method according to claim 7 is implemented.

12. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 6 and / or the method according to claim 7 is implemented.

Images