Development image detection method and system based on feature screening and multi-modal feature fusion

Abstract

The invention discloses a development image detection method and system based on feature screening and multi-modal feature fusion, and the method comprises the steps: obtaining reconstruction features before and after disturbance addition through a variational auto-encoder, and carrying out the hypothesis testing through the reconstruction features before and after disturbance addition, so as to screen physiological data; key physiological data related to development image detection are extracted, then, development images are input into an image classification network to extract morphological features, and image features are obtained; performing standardization conversion on the screened key physiological data to obtain physiological features; and splicing the image features with the screened physiological features, and inputting the spliced features into a development image detection model to complete detection of the development image. Compared with the method of fusing unscreened physiological features by using image features, the method of the invention effectively reduces the interference of useless features in high-dimensional physiological features on the model discrimination capability, and further improves the model performance, thereby realizing more accurate development image detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image detection, and particularly relates to a development image detection method and system based on feature screening and multi-modal feature fusion. BACKGROUND

[0002] Infertility is one of the important problems to be solved in the health field in China. With the wide application of assisted reproductive technology (ART), in vitro fertilization and embryo transfer (IVF-ET) has become an important means to solve the problem of infertility. In order to improve the success rate of embryo transfer, how to accurately detect the development image of the embryo and make pregnancy risk judgment based on the detection result has become one of the key problems in the research and practice of assisted reproduction field.

[0003] At present, the development image detection mainly relies on manual observation of images and subjective judgment of the development state of blastocyst according to Gardner scoring standard. However, this method is difficult to realize efficient, objective and repeatable evaluation. Therefore, it is an urgent need to develop an automatic and intelligent development image detection method.

[0004] In recent years, deep learning technology has achieved remarkable results in the field of medical image analysis. In the development image detection task, there are schemes using convolutional neural network to extract image features to realize the development image detection, but single dependence on image information often cannot fully reflect the real development potential. At the same time, physiological data (such as age, endocrine level, genetic information, etc.) are also proved to be closely related to the risk of pregnancy. However, the high dimension, complex type, and a large amount of missing or redundant information existing in the physiological data bring many challenges to feature screening and fusion. SUMMARY

[0005] The purpose of the present application is to provide a development image detection method and system based on feature screening and multi-modal feature fusion.

[0006] In the first aspect, the present application provides a development image detection method based on feature screening and multi-modal feature fusion, which comprises the following steps: constructing a data set comprising development images and physiological data of different subjects, and performing label processing on the data set; and pre-processing the physiological data and development images in the data set; construct a variational autoencoder, and train the variational autoencoder using physiological data in the dataset; add perturbations to the labels, and input the labels before and after adding the perturbations respectively into the trained variational autoencoder to obtain reconstructed features before and after adding the perturbations; perform hypothesis testing on the reconstructed features before and after adding the perturbations, and filter key physiological data from all the physiological data based on a result of the hypothesis testing; construct physiological features based on the key physiological data of each subject; perform feature extraction on the development images using an image classification network to obtain image features; and splice the physiological features and the image features to obtain fusion features; construct a development image detection model, and train the development image detection model using the fusion features and the labels; obtain a development image to be tested and key physiological data of a subject to be tested, extract physiological features and image features from the key physiological data and the development image respectively, splice the physiological features and the image features, and input the spliced features into the trained development image detection model to obtain a prediction result of the development image, and perform pregnancy risk judgment based on the prediction result.

[0007] Preferably, in the process of training the variational autoencoder, a loss function adopted by the variational autoencoder includes a reconstruction loss and a K-L divergence loss; if the physiological data for training the variational autoencoder belongs to categorical physiological data, a binary cross-entropy loss function is adopted as the reconstruction loss; if the physiological data for training the variational autoencoder belongs to continuous physiological data, a mean square error loss function is adopted as the reconstruction loss.

[0008] Preferably, the hypothesis testing adopts t a test method.

[0009] Preferably, the variational autoencoder includes an encoder, a latent space, and a decoder; the encoder, the latent space, and the decoder are all composed of fully connected layers, and each fully connected layer is provided with a batch normalization layer, a dropout layer, and an activation function.

[0010] Preferably, the development image detection model includes serially connected fully connected layers and activation functions.

[0011] Preferably, the method for pre-processing the physiological data is: performing one-hot encoding on the categorical physiological data in the physiological data, and encoding each bit of the missing categorical physiological data as zero; performing Z-score processing on the continuous physiological data in the physiological data, and assigning a value of zero to the missing continuous physiological data.

[0012] Preferably, the method for pre-processing the development image is: performing random cropping and size adjustment, random vertical flipping, random horizontal flipping, random angle rotation, random brightness and contrast transformation, and random Gaussian noise addition image preprocessing operations on the development image.

[0013] As a preference, a cross-entropy loss function is used as the loss function of the developmental image detection model.

[0014] In a second aspect, the present application provides a developmental image detection system based on feature screening and multi-modal feature fusion, which is used to execute the developmental image detection method described above; the developmental image detection system comprises a data acquisition module, a data preprocessing module, a feature extraction module and a developmental image detection module; the data acquisition module is used to acquire developmental images and physiological data; the data preprocessing module is used to preprocess the physiological data and the developmental images; the feature extraction module is used to extract physiological features and image features from the preprocessed physiological data and developmental images respectively; and the developmental image detection module is used to acquire a detection result of the corresponding developmental image according to the physiological features and the image features.

[0015] In a third aspect, the present application provides a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the developmental image detection method described above.

[0016] The present application has the following beneficial effects: 1. The present application effectively reduces the dimension of the input model and eliminates redundant physiological data by introducing a variational autoencoder and combining a hypothesis testing method for unsupervised learning and statistical analysis of physiological data, effectively screening out key physiological features highly related to developmental image detection from a large amount of physiological data, thereby effectively reducing the interference of useless features in high-dimensional physiological features on the model discrimination ability, further improving the model performance, and thus realizing more accurate developmental image detection.

[0017] 2. The present application combines physiological data and developmental images for developmental image detection, processes the physiological data by using one-hot encoding and Z-score standardization to solve the problems of complex types and missing information in the physiological data, obtains physiological features, uses a convolutional neural network to extract features from the developmental images, obtains image features, and performs cross-modal fusion of the physiological features and the image features, thereby enabling the present application to more comprehensively reflect the pregnancy risk and effectively improve the automatic evaluation ability of the model for the pregnancy risk compared to only relying on developmental images for detection. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The present application is a whole flowchart.

[0019] Figure 2 The present application is a structure diagram of a variational autoencoder.

[0020] Figure 3A network structure diagram for image feature extraction and feature fusion in the present application. DETAILED DESCRIPTION

[0021] The present application is further described below in conjunction with the accompanying drawings.

[0022] In order to have a clearer understanding of the technical features, objectives and effects of the present application, the specific embodiments of the present application are described below in conjunction with the accompanying drawings.

[0023] Embodiment 1 A development image detection system based on feature screening and multi-modal feature fusion, comprising a data acquisition module, a data preprocessing module, a feature extraction module and a development image detection module; the data acquisition module is used to acquire development images and physiological data; the data preprocessing module is used to preprocess the physiological data and the development images; the feature extraction module is used to extract physiological features and image features from the preprocessed physiological data and development images respectively; the development image detection module is used to obtain the detection result of the corresponding development image according to the physiological features and the image features.

[0024] Embodiment 2 As shown in Figure 1 , a development image detection method based on feature screening and multi-modal feature fusion, using the development image detection system in embodiment 1; the development image detection method comprises the following steps: Step one, constructing a data set Acquire development images and physiological data of subjects, and construct a data set; the physiological data includes the age of the subject, the thickness of the endometrium, the level of anti-Mullerian hormone and other hormones, the morphokinetic parameters of proteomics and genomics, the number of cumulus-oocyte complexes, and other obtainable physiological data. Missing physiological data is replaced by NA. The data set is labeled.

[0025] Step two, data preprocessing 2-1. Physiological data preprocessing The physiological data in the data set is standardized; wherein, the category type physiological data in the physiological data is one-hot encoded, and each bit of the missing category type physiological data is encoded as 0; the continuous type physiological data in the physiological data is processed by Z-score, i.e. ; wherein, is the standardized data of each type of physiological data; is the original data; is the data mean; is the data standard deviation. The missing continuous type physiological data is assigned a value of 0, and the dimension of the standardized physiological features is C in , C inThe dimensionality varies with the data set, in this embodiment, C in The physiological data before and after normalization are shown in Tables 1 and 2, respectively.

[0026] Table 1 Physiological data before normalization Table 2 Physiological data after normalization 2-2. Developmental image preprocessing The developmental images were randomly cropped and adjusted to a fixed size of 224x224 pixels, with a scaling ratio of the cropped area between 80% and 100% to simulate observation under different magnifications. The developmental images were horizontally and vertically flipped with a probability of 50% each, and a random rotation operation was performed on the developmental images with a rotation angle between ±45 degrees to enhance the model's recognition ability for images taken at different angles. Random perturbations within a range of ±20% were introduced in the brightness and contrast of the developmental images to simulate the differences in image features under different lighting conditions during acquisition. At the same time, Gaussian noise with a standard deviation of 0.04 was superimposed on the pixel values of the developmental images to enhance the model's robustness to image noise. Finally, the developmental images were normalized to have a uniform mean and standard deviation of pixel value distribution, thereby providing a stable and uniform input feature space for the subsequent feature extraction network. Through the above series of image enhancement and normalization operations, the model's generalization ability for input images was effectively improved, and the dependence on factors such as shooting angle, image clarity, and lighting conditions was reduced, making the trained model more practical and adaptable.

[0027] Step three, feature extraction 3-1. Extracting physiological features A variational autoencoder (VAE) was constructed. As an unsupervised learning model, the variational autoencoder can model the latent space of complex high-dimensional physiological data. Through the encoder, the input features are compressed into low-dimensional latent features, and through the decoder, the features are reconstructed. In this process, the model can learn the latent correlations between different input features, and subsequently identify key physiological data highly correlated with developmental image detection through statistical testing methods.

[0028] As Figure 2As shown, the variational autoencoder includes an encoder, a latent space and a decoder; the encoder, the latent space and the decoder are each composed of a fully connected layer network, and each layer of the fully connected layer network is provided with a batch normalization layer, a Dropout layer and a LeakyReLU activation function. The encoder is responsible for compressing high-dimensional input data into a low-dimensional latent representation space. In this embodiment, the number of nodes of the encoder varies from 524 to 512, from 512 to 256, and from 256 to 64, and the number of nodes can be flexibly configured according to the dimension of the input data and the compression requirement. The latent dimension of the latent space represents the number of hidden variables learned by the variational autoencoder in the encoding process, and different latent dimension settings can be used to explore the compression representation effect of physiological features. The decoder reconstructs the original input features from the latent space, and the number of nodes thereof varies symmetrically with the encoder. In this embodiment, the number of nodes of the decoder varies from 64 to 256, from 256 to 512, and from 512 to 524.

[0029] The variational autoencoder is trained using the preprocessed physiological data; in the training process, the loss function adopted by the variational autoencoder includes a reconstruction loss and a K-L divergence loss. Among them, the reconstruction loss is designed differently for different types of physiological data: for categorical physiological data, a binary cross-entropy loss function is used; for continuous physiological data, a mean square error loss function is used. In this embodiment, the weight of each loss function is set to 1, and by jointly optimizing these two parts of loss, the variational autoencoder can achieve effective reconstruction while maintaining the regularized distribution of latent variables. After training is completed, the labels input to the variational autoencoder are artificially disturbed to introduce disturbance into the variational autoencoder, simulating the influence of label changes on physiological feature expression; the number of re-fitting times is set to 10, and 4 latent space dimensions, i.e., 56, 64, 72 and 80, are set. Under each latent space dimension configuration, the data before the disturbance is retrained, the data after the disturbance is input into the trained variational autoencoder and is re-fitted multiple times, thereby obtaining a series of reconstructed features.

[0030] The reconstructed features output by the model before and after the disturbance are subjected to multiple hypothesis tests to test the significant influence of the disturbed labels on other physiological features, thereby screening out key physiological features highly relevant to the developmental image detection. The specific process is as follows: t The test method screens physiological data closely related to the developmental image detection, and when the significance level pWhen the value is less than 0.05 (p < 0.05), it is considered that the physiological characteristics have statistical significance under the label change in this test, and when statistical significance occurs in at least 8 of 10 times of re-fitting, it is considered that the physiological characteristics have statistical significance under the label change in the latent space; when statistical significance occurs in at least 3 of 4 latent spaces, it is finally determined that the physiological characteristics have strong correlation with the development image detection. The physiological data with strong correlation with the development image detection is used as the key physiological data for subsequent multi-modal fusion and development image detection tasks.

[0031] All the key physiological data are standardized respectively to eliminate the influence of the dimensional difference between different features on model training. The physiological data of each subject is represented as a one-dimensional vector with a dimension of C out The length of the one-dimensional vector is related to the number of key physiological data screened out. In this embodiment, C out 40, which is the physiological feature for subsequent fusion with image features.

[0032] 3-2. Extracting image features As shown in Figure 3 , the image classification network is used to extract features from the preprocessed development image to obtain one-dimensional image features, and the specific process is as follows: The preprocessed development image is input into the image classification network, and the weights pre-trained on the ImageNet dataset are used as the initial weights of the image classification network. First, a 7x7 convolutional layer and a 3x3 max-pooling layer are used to perform preliminary feature extraction and down-sampling on the image. Then, the image is processed by four residual modules in turn, and the number of residual blocks included in the four residual modules is 3, 4, 6, and 3 respectively. With the deepening of the network, the spatial resolution of the image feature map gradually decreases, while the channel dimension gradually increases, obtaining a high-dimensional feature map with a size of 3x3x2048. The high-dimensional feature map is compressed into a one-dimensional image feature vector with a size of 1x1x2048 through global average pooling, which is the expression of the development image in the high-level semantic space, i.e., the image feature.

[0033] In this embodiment, the image classification network uses the ResNet50 network.

[0034] 3-3. Concatenating physiological features and image features to obtain fusion features.

[0035] Step four, constructing a development image detection model; the development image detection model includes a series of fully connected layers and a Softmax activation function. In this embodiment, the input node number of the fully connected layer is 2088, and the output node number is 6. The cross-entropy loss function is used as the loss function of the development image detection model.

[0036] Step five, acquiring the measured development image and the key physiological data of the measured subject, extracting physiological features and image features from the key physiological data and the development image respectively, and inputting the physiological features and the image features into the trained development image detection model after splicing to obtain a detection result of the development image Embodiment 3 A development image detection method based on feature screening and multi-modal feature fusion, which is different from the embodiment 2 in that the development image is an embryo development image developed to the blastocyst stage. The label of the data set is the Gardner score performed on the development image. The non-diagnostic purpose pregnancy risk judgment is performed based on the detection result of the development image detection model. By combining the physiological features and the image features, the detection of the development image can be more comprehensive, thereby reducing the pregnancy risk. At the same time, the detection of the development image by the present application does not violate the ethical rules and does not produce excessive detection cost, and has a wide application range.

[0037] The development image detection model can be used to separately predict different structural parts (inner cell mass, trophoblast ectoderm and blastocyst expansion degree) of the development image, or can uniformly output the detection results of multiple parts. When the detection results of multiple parts are simultaneously predicted, the final output vector is divided into multiple sub-vectors according to the preset dimension, respectively corresponding to the detection results of each part, and the corresponding loss function is calculated for each sub-vector, and finally the sum of each loss term is taken as the total loss.

Claims

1. A development image detection method based on feature screening and multimodal feature fusion, characterized by: The method includes: Construct a dataset including developmental images and physiological data of different subjects, and label the dataset; preprocess the physiological data and developmental images in the dataset; Construct a variational autoencoder and train it using the physiological data in the dataset; add perturbations to the labels and input the labels before and after the perturbations into the trained variational autoencoder to obtain reconstructed features before and after the perturbations; perform hypothesis tests on the reconstructed features before and after the perturbations, and screen key physiological data from all physiological data based on the results of the hypothesis tests; construct physiological features based on the key physiological data of each subject; use an image classification network to extract features from the developmental images to obtain image features; and concatenate the physiological features and image features to obtain fused features. Build a developmental image detection model and train it using fused features and labels; The key physiological data of the developmental images and the subjects under test are obtained, and physiological features and image features are extracted from the key physiological data and developmental images respectively. The physiological features and image features are spliced ​​and input into the trained developmental image detection model to obtain the prediction results of the developmental images.

2. The development image detection method based on feature screening and multimodal feature fusion according to claim 1, characterized in that: During the training process of the variational autoencoder, the loss functions used by the variational autoencoder include reconstruction loss and KL divergence loss; if the physiological data used to train the variational autoencoder is categorical physiological data, the binary cross entropy loss function is used as the reconstruction loss; if the physiological data used to train the variational autoencoder is continuous physiological data, the mean square error loss function is used as the reconstruction loss.

3. The development image detection method based on feature screening and multimodal feature fusion according to claim 1, characterized in that: The hypothesis test uses t Inspection method.

4. The development image detection method based on feature screening and multimodal feature fusion according to claim 1, characterized in that: The variational autoencoder includes an encoder, a latent space and a decoder; the encoder, the latent space and the decoder are all composed of a fully connected layer network, and each fully connected layer network is followed by a batch normalization layer, a dropout layer and an activation function.

5. The method for developmental image detection based on feature screening and multimodal feature fusion according to claim 1, characterized in that: The development image detection model includes a series of fully connected layers and activation functions.

6. The method for developmental image detection based on feature screening and multimodal feature fusion according to claim 1, characterized in that: The method for preprocessing physiological data is as follows: performing one-hot encoding on the categorical physiological data in the physiological data, and encoding each bit of the missing categorical physiological data as zero; performing Z-score processing on the continuous physiological data in the physiological data, and assigning zero to the missing continuous physiological data.

7. The method for developmental image detection based on feature screening and multimodal feature fusion according to claim 1, characterized in that: The method for preprocessing the developmental image is as follows: random cropping and resizing, random vertical flipping, random horizontal flipping, random angle rotation, random brightness and contrast transformation, and random addition of Gaussian noise to the developmental image.

8. The development image detection method based on feature screening and multimodal feature fusion according to claim 1, characterized in that: The cross entropy loss function is used as the loss function of the development image detection model.

9. A development image detection system based on feature screening and multimodal feature fusion, characterized by: Used to perform the developmental image detection method based on feature screening and multimodal feature fusion as described in claim 1; the developmental image detection system includes a data acquisition module, a data preprocessing module, a feature extraction module and a developmental image detection module; the data acquisition module is used to obtain developmental images and physiological data; The data preprocessing module is used to preprocess the physiological data and developmental images; the feature extraction module is used to extract physiological features and image features from the preprocessed physiological data and developmental images respectively; The development image detection module is used to obtain the detection results of the corresponding development image based on physiological characteristics and image features.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory stores a computer program; and the processor executes the development image detection method according to any one of claims 1 to 8.

Images