A pre-training-based pregnancy test record representation learning method

Abstract

The application discloses a kind of pre-training-based pregnancy test record representation learning method, comprising the following steps: S1, the self-attention mechanism in the modification Transformer encoder is constructed pregnancy test record representation model;S2, establish similarity prediction task;S3, establish mask prediction task;S4, establish rationality check task;S5, based on the model pre-training of task described in steps S2-S4, and fine-tuning is carried out to downstream task.Through introducing Transformer encoder to expand time-aware multi-head attention mechanism, irregular time interval can be effectively handled;Three special pre-training tasks are used to effectively train the proposed neural network architecture, and the problems of insufficient data, incomplete data and short sequence are solved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically to a pre-training-based pregnancy examination record representation learning method. Background Art

[0002] Currently, classical time series models are primarily used to solve a specific diagnostic task or address certain unique aspects of electronic health records. This makes it difficult to repurpose these existing methods for the early diagnosis of pregnancy complications or to provide general solutions to address the range of health issues caused by pregnancy complications. To achieve this goal, a fundamental research question is how to derive an effective data representation from prenatal examination data that can capture the key data characteristics of the examination records.

[0003] However, prenatal checkup data is quite complex, and designing effective representation learning methods is not easy. Modeling prenatal checkup data requires addressing at least three major challenges. First, prenatal checkup data changes dynamically at irregular intervals. During pregnancy, physical characteristics (such as weight, foot height, and abdominal circumference) can change significantly. Furthermore, prenatal checkup records correspond to irregularly distributed samples of women's physical characteristics throughout pregnancy. It is difficult to effectively extract and learn time-aware representations from such dynamic, irregular, and unstable prenatal care data. Second, different pregnancy complications often correspond to different factors or indicators. For example, gestational diabetes is more sensitive to the timestamp of prenatal checkup records, while gestational hypertension is more sensitive to checkup records from a specific gestational week. Third, prenatal checkup data is very sparse and even incomplete. Each prenatal checkup only includes specific examination items. For example, examination records are only available when the pregnant woman undergoes a physical examination, and the use of prenatal checkup data is limited by the examination items.

[0004] Therefore, providing a general and robust representation learning method for various medical downstream tasks related to pregnancy complications is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a pre-training-based representation learning method for prenatal examination records; by introducing a Transformer encoder to expand the time-aware multi-head attention mechanism, it can effectively handle irregular time intervals; through three special pre-training tasks, the proposed neural network architecture is effectively trained to solve the problems of insufficient data, incomplete data and short sequences.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for learning representations of pregnancy examination records based on pre-training, comprising the following steps:

[0008] S1. Modify the self-attention mechanism in the Transformer encoder to build a representation model for pregnancy test records;

[0009] S2, establish similarity prediction task;

[0010] S3, establish the cover prediction task;

[0011] S4. Establish rationality check tasks;

[0012] S5. Pre-train the model based on the tasks described in steps S2-S4 and fine-tune the downstream tasks.

[0013] Preferably, the step S1 specifically includes:

[0014] Create a data sequence<C,τ> , suppose each pregnancy checkup is a visit c, and each check result is e, then visit c is the vector set of check result e, and data sequence C is the collection of visits c, C=(c1, c2, ..., c n ), visit c corresponds to a gestational week τ i , then the gestational age set τ=(τ1,τ2,…,τ n );

[0015] Add virtual visits to the data series * , the virtual visit c * The virtual gestational age is τ * =τ T +1;

[0016] Given an access vector c in the data sequence C t , use full connection to map it into a high-dimensional space:

[0017] x t =W x ×c t +b x

[0018] Among them, x t ∈R h , b x ∈R h , are all learnable parameters;

[0019] The results are stacked to get the embedding matrix X = (x1, ..., x T , x * );

[0020] Use position coding to encode the position information, use gestational age τ instead of position information, and add the generated position coding to x t middle,

[0021] ut =PE(τ t )

[0022] p t =x t +u t

[0023] PE represents the position encoding function in Transformer, and superimposing all the vectors of the patients yields P = (p1, ..., p T , p * );

[0024] The self-attention mechanism of the standard Transformer encoder is:

[0025]

[0026] The standard Transformer encoder cannot reflect the size of the time interval, so a fully connected layer is introduced to capture the time span information. The self-attention mechanism of the modified Transformer encoder is:

[0027]

[0028] Among them, W τ ∈R 1×h ,|·| is the absolute value of the element, use|u i -u j |Incorporate time interval information into time awareness considerations;

[0029] The input of TSA includes query Q, secret key K and value V, then the multi-head attention mechanism TSA is expressed as:

[0030] TSA(Q,K,V,W τ ,τ)=softmax(A (T) )×V

[0031] The time-aware multi-head attention mechanism concatenates multiple individual TSAs and inputs them into a fully connected network:

[0032]

[0033] G=W o ×(G (1) ||G (2) ||…||G (n) )

[0034] Among them, “||” represents vector splicing, W o ∈R h×nh, using layer normalization and residual connection, a feed-forward layer processes the hidden state h of each visit separately t , specifically:

[0035] s t =W2×ReLU(W1×g t +b1)+b2

[0036] Where W1, W2∈R h×h ,b1,b2∈R h , ReLU(x)=max(0,x), and by normalizing the residual connection and layer, we get the final representation of the pregnancy test record:

[0037] S=(s1,...,s T , s * ).

[0038] Preferably, the specific steps of step S2 are:

[0039] S21. Calculate the Euclidean distance between each pair of the last week's prenatal checkup records of all pregnant women in the dataset, i.e.

[0040] And the smaller the value, the higher the similarity;

[0041] S22, the most similar 15% are positive samples, and the least similar 15% are negative samples, then the original number of samples N becomes

[0042] S23. Based on the Siamese network, the samples are mapped to a latent space, where samples with close feature distances have close semantic distances, as given by and Represents a sample pair, and measures their semantic distance based on the Euclidean distance Then the twin network is in N p The loss function on is defined as:

[0043]

[0044] Among them, z i are positive and negative samples, z i =1 is a positive sample, z i =0 is a negative sample, N p is the sample logarithm, d i is the Euclidean distance, and m is the preset parameter.

[0045] Preferably, the step S3 specifically includes:

[0046] S31, randomly block 30% of c* access;

[0047] S32, set Mask access The corresponding hidden state of is used to predict the source inspection record based on a multi-layer perceptron MLP with ReLU activation:

[0048]

[0049] The mean square error MSE is used as the optimization target:

[0050]

[0051] in, For the set of all masked visits, for size.

[0052] Preferably, the step S4 specifically includes:

[0053] S41, 50% of the sequences are selected as negative samples, and then 50%-75% of the visits of these sequences are randomly selected and replaced with visits of other sequences, and the other 50% are positive samples;

[0054] S42. Use a multi-layer perception mechanism with ReLU as the activation function to predict whether it is reasonable:

[0055]

[0056] in, To map the output to the activation function of (0, 1), let r be the rationality label, and use the cross entropy loss function for target optimization:

[0057]

[0058] Among them, IVr is the sample size.

[0059] Preferably, the step S5 specifically includes:

[0060] The model is trained on three pre-training tasks, and the final pre-training loss function is defined as:

[0061] L p =λ1L s +λ2L m +(1-λ1-λ2)L r

[0062] Among them, λ1 and λ2 are hyperparameters for balancing the three pre-training tasks;

[0063] For classification tasks, a fully connected layer is used for activation, namely:

[0064]

[0065] Among them, W c ∈R 1×h , b c ∈R 1 ;

[0066] Let y represent the label, and then use the cross entropy loss as the optimization target:

[0067]

[0068] Where N is the number of pregnant women;

[0069] For regression tasks, based on the fully connected layer:

[0070]

[0071] in, N f is the number of regression targets;

[0072] Using mean square error MSE as the optimization target, we can get:

[0073]

[0074] Where N is the number of pregnant women.

[0075] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for learning the representation of pregnancy examination records based on pre-training; by introducing the Transformer encoder to expand the time-aware multi-head attention mechanism, irregular time intervals can be effectively processed; and the proposed neural network architecture is effectively trained through three special pre-training tasks to solve the problems of insufficient data, incomplete data and short sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0077] Figure 1 The accompanying drawing is a schematic diagram of the method flow structure provided by the present invention.

[0078] Figure 2 The accompanying drawing is a schematic diagram of the model structure provided by the present invention.

[0079] Figure 3 The accompanying drawing is a schematic diagram of the results provided by the present invention. DETAILED DESCRIPTION

[0080] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0081] The embodiment of the present invention discloses a method for learning representations of pregnancy examination records based on pre-training, comprising the following steps:

[0082] S1. Modify the self-attention mechanism in the Transformer encoder to build a representation model for pregnancy test records;

[0083] S2, establish similarity prediction task;

[0084] S3, establish the cover prediction task;

[0085] S4. Establish rationality check tasks;

[0086] S5. Pre-train the model based on the tasks described in steps S2-S4 and fine-tune the downstream tasks.

[0087] To further optimize the above technical solution, step S1 specifically includes:

[0088] Create a data sequence<C,τ> , suppose each pregnancy checkup is a visit c, and each check result is e, then visit c is the vector set of check result e, and data sequence C is the collection of visits c, C=(c1, c2, ..., c n ), visit c corresponds to a gestational week τ i , then the gestational age set τ=(τ1,τ2,…,τ n );

[0089] Add virtual visits to the data series * , the virtual visit c * The virtual gestational age is τ * =τ T +1;

[0090] Given an access vector in the data sequence C ct , use full connection to map it into a high-dimensional space:

[0091] x t =W x ×c t +b x

[0092] Among them, x t ∈R h, b x ∈R h , are all learnable parameters;

[0093] The results are stacked to get the embedding matrix X = (x1, ..., x T , x * );

[0094] Use position coding to encode the position information, use gestational age τ instead of position information, and add the generated position coding to x t middle,

[0095] u t =PE(τ t )

[0096] p t =x t +u t

[0097] PE represents the position encoding function in Transformer, and superimposing all the vectors of the patients yields P = (p1, ..., p T , p * );

[0098] The self-attention mechanism of the standard Transformer encoder is:

[0099]

[0100] The standard Transformer encoder cannot reflect the size of the time interval, so a fully connected layer is introduced to capture the time span information. The self-attention mechanism of the modified Transformer encoder is:

[0101]

[0102] Among them, W τ ∈R 1×h ,|·| is the absolute value of the element, use|u i -u j |Incorporate time interval information into time awareness considerations;

[0103] The input of TSA includes query Q, secret key K and value V, then the multi-head attention mechanism TSA is expressed as:

[0104] TSA(Q,K,V,W τ ,τ)=softmax(A (T) )×V

[0105] The time-aware multi-head attention mechanism concatenates multiple individual TSAs and inputs them into a fully connected network:

[0106]

[0107] G=W o ×(G (1) ||G (2) ||…||G (n) )

[0108] Among them, “||” represents vector splicing, W o ∈R h×nh , using layer normalization and residual connection, a feed-forward layer processes the hidden state hhht of each visit separately, specifically:

[0109] s t =W2×ReLU(W1×g t +b1)+b2

[0110] Where W1, W2∈R h×h ,b1,b2∈R h , ReLU(x)=max(0,x), and by normalizing the residual connection and layer, we get the final representation of the pregnancy test record:

[0111] S=(s1,...,s T , s * ).

[0112] To further optimize the above technical solution, the specific steps of step S2 are:

[0113] S21. Calculate the Euclidean distance between each pair of the last week's prenatal checkup records of all pregnant women in the dataset, i.e.

[0114] And the smaller the value, the higher the similarity;

[0115] S22, the most similar 15% are positive samples, and the least similar 15% are negative samples, then the original number of samples

[0116] S23. Based on the Siamese network, the samples are mapped to a latent space, where samples with close feature distances have close semantic distances, as given by and Represents a sample pair, and measures their semantic distance based on the Euclidean distance Then the twin network is in N p The loss function on is defined as:

[0117]

[0118] Among them, z iare positive and negative samples, z i =1 is a positive sample, z i =0 is a negative sample, N p is the sample logarithm, d i is the Euclidean distance, and m is the preset parameter.

[0119] To further optimize the above technical solution, step S3 specifically includes:

[0120] S31, randomly block 30% of c* access;

[0121] S32, set Mask access The corresponding hidden state of is used to predict the source inspection record based on a multi-layer perceptron MLP with ReLU activation:

[0122]

[0123] The mean square error MSE is used as the optimization target:

[0124]

[0125] in, For the set of all masked visits, for size.

[0126] To further optimize the above technical solution, step S4 specifically includes:

[0127] S41, 50% of the sequences are selected as negative samples, and then 50%-75% of the visits of these sequences are randomly selected and replaced with visits of other sequences, and the other 50% are positive samples;

[0128] S42. Use a multi-layer perception mechanism with ReLU as the activation function to predict whether it is reasonable:

[0129]

[0130] in, To map the output to the activation function of (0, 1), let r be the rationality label, and use the cross entropy loss function for target optimization:

[0131]

[0132] Among them, N r is the sample size.

[0133] To further optimize the above technical solution, step S5 specifically includes:

[0134] The model is trained on three pre-training tasks, and the final pre-training loss function is defined as:

[0135] L p =λ1L s +λ2L m +(1-λ1-λ2)L r

[0136] Among them, λ1 and λ2 are hyperparameters for balancing the three pre-training tasks;

[0137] For classification tasks, a fully connected layer is used for activation, namely:

[0138]

[0139] Among them, W c ∈R 1×h , b c ∈R 1 ;

[0140] Let y represent the label, and then use the cross entropy loss as the optimization target:

[0141]

[0142] Where N is the number of pregnant women;

[0143] For regression tasks, based on the fully connected layer:

[0144]

[0145] in, N f is the number of regression targets;

[0146] Using mean square error MSE as the optimization target, we can get:

[0147]

[0148] Where N is the number of pregnant women.

[0149] During the training process, we first pre-trained the time-aware transformer with three pre-training tasks. Once our model is pre-trained, we use task-specific losses to fine-tune the time-aware transformer and prediction components to achieve better performance. Compared with traditional EHR data modeling methods, RAPT has the following advantages. First, the model can handle various problems such as insufficient data and incomplete data. Second, the proposed model can handle several problems without the need for additional components. Third, the model is not designed for a specific task. It provides a general solution to health problems.

[0150] Finally, we compared the pre-training tasks with those in other fields. In natural language processing (NLP), common pre-training tasks include masked language modeling (MLM), next sentence prediction (NSP), replacement token detection (RTD), and sentence order prediction (SOP). We followed the MLM task and modified it to apply to EHR data. For other tasks, NSP and RTD are not applicable to EHR data, considering that the examination records of healthy pregnant women are similar. For healthy pregnant women, except for examination records such as weight, which rarely have obvious change trends, other examination records remain stable throughout the pregnancy, so SOP is not used for EHR data. In computer vision (CV), the mainstream pre-training task is instance recognition. This task regards each sample as a class. Obviously, it is not applicable to EHR data, but we proposed a similarity prediction task, which has similar effects to instance recognition.

[0151] In predicting gestational diabetes, the method proposed in the present invention improves the AUC score by 5.4%, the F1 score by 5.5%, and the Accuracy by 6.0% compared with the existing methods.

[0152] In predicting gestational hypertension, the method proposed in the present invention improves the AUC score by 0.6%, the F1 score by 0.3%, and the Accuracy by 0.7% compared with existing methods.

[0153] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0154] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pre-training-based learning method for representing pregnancy examination records, characterized in that: The following steps are involved: S1. Modify the self-attention mechanism in the Transformer encoder to build a representation model for pregnancy test records; The standard Transformer encoder cannot reflect the size of the time interval, so a fully connected layer is introduced to capture the time span information. The self-attention mechanism of the modified Transformer encoder is: Among them, W τ ∈R 1×h ,|·| is the absolute value of the element, use|u i -u j |Incorporate time interval information into time awareness considerations; The input of TSA includes query Q, secret key K and value V, then the multi-head attention mechanism TSA is expressed as: TSA(Q,K,V,W τ ,τ)=softmax(A (T) )×V The time-aware multi-head attention mechanism concatenates multiple individual TSAs and inputs them into a fully connected network: G=W o ×(G (1) ||G (2) ||…||G (n) ) Among them, "||" represents vector splicing, W o ∈R h×nh , using layer normalization and residual connection, a feed-forward layer processes the hidden state h of each visit separately t , specifically: s t =W2×ReLU(W1×g t +b1)+b2 Where W1, W2∈R h×h ,b1,b2∈R h , ReLU(x)=max(0,x), and by normalizing the residual connection and layer, we get the final representation of the pregnancy test record: S=(s1,...,s T ,s * ); S2, establish similarity prediction task; S3, establish the cover prediction task; S4. Establish rationality check tasks; S5. Pre-train the model based on the tasks described in steps S2-S4 and fine-tune the downstream tasks.

2. The method for learning the representation of pregnancy examination records based on pre-training according to claim 1, characterized in that: The step S1 specifically includes: Create a data sequence<C,τ> , suppose each pregnancy checkup is a visit c, and each check result is e, then visit c is the vector set of check result e, and data sequence C is the collection of visits c, C=(c1, c2, ..., c n ), visit c corresponds to a gestational week τ i , then the gestational age set τ=(τ1,τ2,…,τ n ); Add virtual visits to the data series * , the virtual visit c * The virtual gestational age is τ * =τ T +1; Given an access vector c in the data sequence C t , use full connection to map it into a high-dimensional space: x t =W x ×c t +b x Among them, x t ∈R h , b x ∈R h , are all learnable parameters; The results are stacked to get the embedding matrix X = (x1, ..., x T , x * ); Use position coding to encode the position information, use gestational age τ instead of position information, and add the generated position coding to x t middle, u t =F(τ t ) p t =x t +u t PE represents the position encoding function in Transformer, and superimposing all the vectors of the patients yields P = (p1, ..., p T , p * ); The self-attention mechanism of the standard Transformer encoder is:

3. The method for learning the representation of pregnancy examination records based on pre-training according to claim 1, characterized in that: The specific steps of step S2 are: S21. Calculate the Euclidean distance between each pair of the last week's prenatal checkup records of all pregnant women in the dataset, i.e. And the smaller the value, the higher the similarity; S22, the most similar 15% are positive samples, and the least similar 15% are negative samples, then the original number of samples N becomes S23. Based on the Siamese network, the samples are mapped to a latent space, where samples with close feature distances have close semantic distances, as given by and Represents a sample pair, and measures their semantic distance based on the Euclidean distance Then the twin network is in N p The loss function on is defined as: Among them, z i are positive and negative samples, z i =1 is a positive sample, z i =0 is a negative sample, N p is the sample logarithm, d i is the Euclidean distance, and m is the preset parameter.

4. The method for learning the representation of pregnancy examination records based on pre-training according to claim 1, characterized in that: The step S3 specifically includes: S31, randomly shield 30% of c * access; S32, set Mask access The corresponding hidden state of is used to predict the source inspection record based on a multi-layer perceptron MLP with ReLU activation: The mean square error MSE is used as the optimization target: in, For the set of all masked visits, for size.

5. The method for learning pregnancy examination record representation based on pre-training according to claim 1, characterized in that: The step S4 specifically includes: S41, 50% of the sequences are selected as negative samples, and then 50%-75% of the visits of these sequences are randomly selected and replaced with visits of other sequences, and the other 50% are positive samples; S42. Use a multi-layer perception mechanism with ReLU as the activation function to predict whether it is reasonable: in, To map the output to the activation function of (0, 1), let r be the rationality label, and use the cross entropy loss function for target optimization: Among them, N r is the sample size.

6. The method for learning pregnancy examination record representation based on pre-training according to claim 1, characterized in that: The step S5 specifically includes: The model is trained on three pre-training tasks, and the final pre-training loss function is defined as: L p =λ1L s +λ2L m +(1-λ1-λ2)L r Among them, λ1 and λ2 are hyperparameters for balancing the three pre-training tasks; For classification tasks, a fully connected layer is used for activation, namely: Among them, W c ∈R 1×h , b c ∈R 1 ; Let y represent the label, and then use the cross entropy loss as the optimization target: Where N is the number of pregnant women; For regression tasks, based on the fully connected layer: in, N f is the number of regression targets; Using mean square error MSE as the optimization target, we can get: Where N is the number of pregnant women.

Images