A method and system for classifying and evaluating oocyte toxicity based on machine learning

Abstract

The present invention relates to a method for classifying and evaluating oocyte toxicity based on machine learning, comprising: obtaining an oocyte image; processing the oocyte image through a neural network segmentation model of a U-shaped structure to obtain key areas in the oocyte image; wherein the key areas include polar body areas, oocyte cytoplasm areas, oocyte outer ring areas and oocyte inner ring areas; calculating key features of oocytes in oocyte images based on key areas and mathematical constraint algorithms; wherein the key features are determined based on expert experience knowledge; extracting local features of oocytes in oocyte images based on a residual neural network model; mapping key features and local features to the same feature domain based on an adaptive attention network; and determining the category of oocytes in oocyte images based on features of the same feature domain. In the method, the entire process is automated and integrated, greatly improving analysis efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of medical artificial intelligence and medical image recognition technology, and in particular to an oocyte toxicity classification evaluation method and system based on machine learning. Background Art

[0002] In the field of medical artificial intelligence and medical image recognition, the quality evaluation of oocytes has always been an indispensable process in the field of assisted reproduction. During the maturation and development of oocytes, the quality of mature oocytes is reduced due to the influence of the environment or its own gene expression. Studies have shown that there is a correlation between oocytes of different morphologies and the quality of oocytes. Oocyte morphological evaluation is an important means to evaluate the quality of oocytes. Traditional oocyte evaluation mainly relies on observation under a microscope and manual interpretation by experts. This method is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in instability and errors in the analysis results. In addition, due to the complex structure of oocytes, subtle morphological changes are often difficult to capture by the naked eye, further increasing the difficulty of analysis. With the development of deep learning technology, some medical image recognition methods based on neural networks have begun to be applied to the analysis of oocytes. However, most of the existing methods only focus on the extraction of global features, ignoring the importance of local features, resulting in poor results in the analysis of subtle morphological changes in oocytes. At the same time, these methods have failed to make full use of expert experience and knowledge, making the analysis results lack a certain degree of interpretability and credibility. Summary of the invention

[0003] In order to solve the above-mentioned problems in the prior art, the present invention provides a method and system for classifying and evaluating oocyte toxicity based on machine learning.

[0004] In a first aspect, an embodiment of the present application provides a method for classifying and evaluating oocyte toxicity based on machine learning, characterized in that it includes: acquiring an oocyte image; processing the oocyte image through a U-shaped neural network segmentation model to acquire key areas in the oocyte image; wherein the key areas include a polar body region, an oocyte cytoplasm region, an oocyte outer ring region, and an oocyte inner ring region; calculating key features of the oocyte in the oocyte image based on the key areas and a mathematical constraint algorithm; wherein the key features are determined based on expert experience knowledge; extracting local features of the oocyte in the oocyte image based on a residual neural network model; based on an adaptive attention network, mapping the key features and the local features to the same feature domain; determining the category of the oocyte in the oocyte image based on features of the same feature domain; wherein the category characterizes whether the oocyte is toxic.

[0005] In an optional implementation of the first aspect, the oocyte image is obtained by collecting images of mouse oocytes using an inverted phase contrast microscope; the oocyte image is stored in a tif or tiff format, and the size of the oocyte image is 680*680.

[0006] In an optional implementation of the first aspect, the oocyte image is processed by a U-shaped neural network segmentation model to obtain a key area in the oocyte image, including: inputting the oocyte image into the U-shaped neural network segmentation model, encoding the oocyte image through a coding layer; decoding the encoded image through a decoding layer to obtain the key area in the oocyte image; wherein the low-level features obtained by the coding layer are passed to a decoder at the same level in the decoding layer; the deep features decoded in the decoding layer are reversely passed to an encoder at the same level in the coding layer; in the coding layer, each convolutional layer repeatedly uses two 3×3 unfilled convolutional layers. A convolution operation is performed and connected in series with the deep features from the decoding layer. A linear correction unit, a global maximum pooling operation with a size of 2×2, and a downsampling operation with a compensation of 2 are connected after each convolution layer; wherein, in the decoding layer, each layer is an iterative upsampling operation of the shallow features, each deconvolution layer performs a deconvolution operation with a size of 2×2, halving the number of channels, and then connected in series with the shallow features from the encoding layer, and then two convolution operations with a convolution kernel size of 3×3 are performed, and each deconvolution layer is followed by a linear correction unit; the last convolution layer adds a convolution operation with a size of 1×1 to map each component feature back to the size of the oocyte image.

[0007] In an optional implementation of the first aspect, the loss function of the U-shaped neural network segmentation model is obtained by weighted calculation of a pixel-level softmax function and a cross entropy loss function.

[0008] In an optional embodiment of the first aspect, the key features include at least the polar body area, the cytoplasm area, the total area inside the inner ring of the zona pellucida, the total area inside the outer ring of the zona pellucida, the perivitelline space, the oocyte diameter and the zona pellucida thickness.

[0009] In an optional embodiment of the first aspect, before inputting the oocyte image into the residual neural network model, the method further includes: preprocessing the original oocyte image to obtain the oocyte image; wherein the preprocessing process includes: increasing the brightness, contrast, boundary information and key point information of the original oocyte image based on a histogram equalization algorithm; normalizing the original oocyte image based on a set mean and a set variance; wherein the set mean is [0.485, 0.456, 0.406] and the set variance is [0.229, 0.224, 0.225].

[0010] In an optional implementation of the first aspect, the residual neural network model includes: a convolutional layer module and a residual module; the convolutional layer module includes two two-dimensional convolutional layers as adaptive image feature extractors, and extracts shallow image features through the convolution process of the convolutional layer module. The model setting parameters use a kernel size of 3×3 and a step size of 2, and each convolution layer is followed by a batch normalization layer and a linear correction unit; the residual module includes three different types of residual blocks, all of which have a kernel size of 3×3 and a step size of 2, and the sizes of the convolution kernels are set to different sizes.

[0011] In an optional implementation of the first aspect, the method further includes: performing salient segment visualization on the features processed by the last residual block based on back-propagation visualization methods Grad-CAM and GuidedGrad-CAM.

[0012] In an optional implementation of the first aspect, the adaptive attention network is an adaptive attention network for tabular data.

[0013] In a second aspect, an embodiment of the present application provides an oocyte toxicity classification and evaluation device based on machine learning, comprising: an acquisition module for acquiring an oocyte image; a processing module for processing the oocyte image through a U-shaped neural network segmentation model to acquire key areas in the oocyte image; wherein the key areas include polar body areas, oocyte cytoplasm areas, oocyte outer ring areas and oocyte inner ring areas; a calculation module for calculating key features of the oocyte in the oocyte image based on the key areas and a mathematical constraint algorithm; wherein the key features are determined based on expert experience knowledge; an extraction module for extracting local features of the oocyte in the oocyte image based on a residual neural network model; a mapping module for mapping the key features and the local features to the same feature domain based on an adaptive attention network; a determination module for determining the category of the oocyte in the oocyte image based on features of the same feature domain; wherein the category characterizes whether the oocyte is toxic.

[0014] The beneficial effects of the present invention include:

[0015] (1) The existing technical solutions often lack a streamlined analysis method in the oocyte analysis process. Specifically, the existing analysis methods are usually carried out step by step and manually, involving multiple links such as segmentation, feature calculation and recognition, and these links lack close connection and automated integration. This non-streamlined analysis method is not only inefficient, but also easily affected by human factors, resulting in instability and deviation of the analysis results. In addition, feature extraction is often based on human experience and domain knowledge, and it is difficult to cover all possible feature combinations, resulting in incomplete extracted features, affecting the performance and generalization ability of the model. The present invention combines deep learning with expert experience and knowledge to realize a process-based automated analysis method starting from the raw data of oocytes, automatically identifying key areas through a neural network segmentation method based on a U-shaped structure, then constructing a mathematical formula based on empirical knowledge to automatically calculate key features, and finally using a feature back-end fusion classification method to automatically complete the model downstream tasks. The entire process is automated and integrated, greatly improving the efficiency and accuracy of analysis.

[0016] (2) The present invention is different from the traditional oocyte data analysis method. It can use the real-time measured oocyte image data as input to complete a series of downstream tasks of the oocyte, making real-time discrimination information feedback possible and providing a theoretical basis for the development of related portable devices.

[0017] (3) The framework of the present invention is based on deep learning (local feature extraction) and expert experience features (global feature extraction), coupled with the concept of attention mechanism and multimodal fusion. Compared with the traditional machine learning feature engineering model and traditional deep learning methods, it is not bound by existing human experience but does not rely entirely on simple data operations. It can extract more features that are consistent with the current scene distribution based on the characteristics of the data and can achieve better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of the steps of a method for classifying and evaluating oocyte toxicity based on machine learning provided in an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a process for obtaining an oocyte image provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a neural network segmentation model with a U-shaped structure provided by an embodiment of the present invention;

[0021] Figure 4 An overall system architecture diagram of a method for classifying and evaluating oocyte toxicity based on machine learning provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of a convolutional layer module provided by an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of a residual module provided by an embodiment of the present invention;

[0024] Figure 7 A schematic diagram of an adaptive attention network provided by an embodiment of the present invention;

[0025] Figure 8 The first experimental data graph provided by the embodiment of the present invention;

[0026] Fig. 9 A second experimental data graph provided by an embodiment of the present invention;

[0027] Fig.10 A third experimental data diagram provided by an embodiment of the present invention;

[0028] Fig.11 The fourth experimental data diagram provided by the embodiment of the present invention;

[0029] Fig.12 The fifth experimental data diagram provided by the embodiment of the present invention;

[0030] Fig.13A module block diagram of an oocyte toxicity classification and evaluation device based on machine learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0032] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context.

[0035] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0037] See also Figure 1 The embodiment of the present application provides an oocyte toxicity classification and evaluation method based on machine learning, including: steps 101 to 106.

[0038] Step 101: Acquire an oocyte image.

[0039] Step 102: Processing the oocyte image through a U-shaped neural network segmentation model to obtain key areas in the oocyte image; wherein the key areas include the polar body area, the oocyte cytoplasm area, the oocyte outer ring area and the oocyte inner ring area.

[0040] Step 103: Calculate key features of the oocyte in the oocyte image based on the key region and the mathematical constraint algorithm; wherein the key features are determined according to expert experience knowledge.

[0041] That is, here, the corresponding key features are extracted according to the setting of expert experience.

[0042] Step 104: extracting local features of the oocyte in the oocyte image based on the residual neural network model.

[0043] Step 105: Based on the adaptive attention network, the key features and local features are mapped to the same feature domain.

[0044] Step 106: Determine the category of the oocyte in the oocyte image based on the features in the same feature domain.

[0045] That is, by mapping key features and local features to the same feature domain, different downstream classification tasks can be performed.

[0046] Among them, a downstream classification task is used to determine whether the oocyte is toxic, that is, the determined category characterizes whether the oocyte is toxic.

[0047] Optionally, downstream classification tasks may also include, but are not limited to, identifying oocytes containing PFOS, F-53B, non-poisoned, PFHxS, PFOA, SCCP (all of the above abbreviations are chemical substances).

[0048] In summary, the beneficial effects of the oocyte toxicity classification evaluation method based on machine learning provided in the embodiments of the present invention include:

[0049] (1) The existing technical solutions often lack a streamlined analysis method in the oocyte analysis process. Specifically, the existing analysis methods are usually carried out step by step and manually, involving multiple links such as segmentation, feature calculation and recognition, and these links lack close connection and automated integration. This non-streamlined analysis method is not only inefficient, but also easily affected by human factors, resulting in instability and deviation of the analysis results. In addition, feature extraction is often based on human experience and domain knowledge, and it is difficult to cover all possible feature combinations, resulting in incomplete extracted features, affecting the performance and generalization ability of the model. The present invention combines deep learning with expert experience and knowledge to realize a process-based automated analysis method starting from the raw data of oocytes, automatically identifying key areas through a neural network segmentation method based on a U-shaped structure, then constructing a mathematical formula based on empirical knowledge to automatically calculate key features, and finally using a feature back-end fusion classification method to automatically complete the model downstream tasks. The entire process is automated and integrated, greatly improving the efficiency and accuracy of analysis.

[0050] (2) The present invention is different from the traditional oocyte data analysis method. It can use the real-time measured oocyte image data as input to complete a series of downstream tasks of the oocyte, making real-time discrimination information feedback possible and providing a theoretical basis for the development of related portable devices.

[0051] (3) The framework of the present invention is based on deep learning (local feature extraction) and expert experience features (global feature extraction), coupled with the concept of attention mechanism and multimodal fusion. Compared with the traditional machine learning feature engineering model and traditional deep learning methods, it is not bound by existing human experience but does not rely entirely on simple data operations. It can extract more features that are consistent with the current scene distribution based on the characteristics of the data and can achieve better performance.

[0052] Optionally, the oocyte image is obtained by collecting images of mouse oocytes using an inverted phase contrast microscope; the oocyte image is stored in a tif or tiff format, and the size of the oocyte image is 680*680.

[0053] For details, please refer to Figure 2 The oocyte images are from oocytes after in vitro maturation culture of mice. The specific protocol is as follows: 6-8 week old sexually mature female Kunming mice were injected with 10 IU pregnant mare serum gonadotropin for superovulation. The mice were dissected 48 hours later, and primary oocytes in the germinal vesicle stage were isolated from the ovaries and placed in preheated culture medium for in vitro maturation culture. After 16 hours of culture, images of the oocytes were immediately collected.

[0054] An inverted phase contrast microscope was used to collect images of mouse oocytes. This phase contrast system uses the differences in refractive index and thickness between different structural components to convert the optical path difference through different parts of the cell into amplitude difference, thereby improving the contrast between various structures, presenting a stronger sense of three-dimensionality, and making the oocyte structure clearly visible, so as to collect higher quality images. In the experiment, oocyte samples after in vitro maturation and culture were placed under a microscope, adjusted to phase contrast mode, and all shooting parameters were saved to collect images of all oocytes. The collected data were saved in the form of tif / tiff images and 680*680 numerical data, including information such as morphological characteristics, size, and shape of oocytes. This acquisition method can provide high-quality mouse oocyte data, providing a reliable basis for subsequent analysis and research.

[0055] Optionally, the oocyte image is processed by a U-shaped neural network segmentation model to obtain key areas in the oocyte image, including: inputting the oocyte image into the U-shaped neural network segmentation model, encoding the oocyte image through the encoding layer; decoding the encoded image through the decoding layer to obtain key areas in the oocyte image; wherein the low-level features obtained by the encoding layer are passed to a decoder at the same level in the decoding layer; and the deep features decoded in the decoding layer are reversely passed to an encoder at the same level in the encoding layer. In the encoding layer, each convolution layer repeatedly uses two 3×3 unpadded convolution operations and is connected in series with the deep features from the decoding layer. Each convolution layer is connected to a linear rectification unit, a global maximum pooling operation of size 2×2, and a downsampling operation with a compensation of 2; in the decoding layer, each layer is an iterative upsampling operation of shallow features. Each deconvolution layer performs a deconvolution operation of size 2×2 to halve the number of channels, and then is connected in series with the shallow features from the encoding layer, and then performs two convolution operations with a convolution kernel size of 3×3. Each deconvolution layer is followed by a linear rectification unit; the last convolution layer adds a convolution operation of size 1×1 to map each component feature back to the size of the oocyte image.

[0056] For details, please refer to Figure 3 The segmentation task of the embodiment of the present invention includes four functions: oocyte outer circle recognition, oocyte inner circle recognition, oocyte cytoplasm recognition, and oocyte polar body recognition. Figure 3The overall architecture of the oocyte key area image segmentation shown in the figure is based on the U-type encoding-decoding neural network segmentation model, and is obtained by supervised training based on expert prior knowledge annotation. In the embodiment of the present invention, the low-level features of the encoding layer are passed to the decoding layer to prevent the loss of information, and the deep features of the decoding are reversely passed from the decoder to the encoder of the same level during the decoding process. The encoder can combine the features of the decoding level with the original input. While further improving the model capability, it does not improve the additional levels and parameter amounts, and can ensure the efficiency of the model use. The specific operations are as follows:

[0057] S1. Encode the original oocyte image. Each convolution layer of the image encoding operation repeatedly uses two 3×3 unfilled convolution operations and connects them in series with the deep features from the decoding process. After that, each convolution is followed by a linear rectification unit, a global maximum pooling operation of size 2×2, and a downsampling operation with a compensation of 2. In each downsampling step, the number of channels of the feature will be doubled.

[0058] S2. Decode the encoded image. Each layer in the decoding process is an iterative upsampling operation of the shallow features. Each convolution layer performs a 2×2 deconvolution operation to halve the number of channels, and then concatenates with the shallow features from the encoding process. After that, two convolution operations with a kernel size of 3×3 are performed, and each convolution is followed by a linear rectification unit. Since the boundary information is lost in each convolution process, we add a 1×1 convolution operation in the last convolution layer to map each component feature back to the size of the original image.

[0059] S3. Model training loss function selection. The model training process adopts the classic five-fold cross-training mode and is tested on an independent test set. The loss function is a weighted calculation of the pixel-level softmax function and the cross entropy loss function, as shown below:

[0060]

[0061] w(x)=wc(x)+W0*exp(-(d1(x)+d(x)) / 2σ2);

[0062] Loss=∑ x∈l w(x)log(Softmax l(x) (x)

[0063] Among them, act k(x) is the activation value of the activation function of the convolution channel k at the pixel position x, K is the total number of categories at the pixel level, and in the embodiment of the present invention, its number is 2, and w(x) is the pixel importance function given in the model training process, where w c (x) is the initial defined weight distribution of the balanced classification weight, d1 is the distance from the pixel to the nearest classification boundary, d2 is the distance from the pixel to the second nearest classification boundary, and l is the true label of each pixel in the image.

[0064] That is, the loss function of the U-shaped neural network segmentation model is obtained by weighted calculation of the pixel-level softmax function and the cross entropy loss function.

[0065] Optionally, the key features include at least the polar body area, the cytoplasm area, the total area inside the inner ring of the zona pellucida, the total area inside the outer ring of the zona pellucida, the perivitelline space, the oocyte diameter and the zona pellucida thickness.

[0066] Based on the four key regions of the oocyte generated by the segmentation function, we extracted seven key features based on clinical expert experience, as follows:

[0067] S1, polar body area (um 2 ). The specific calculation formula is as follows:

[0068]

[0069] o1 is the polar body region automatically segmented by the model, where m is the height of region o1 (i.e. the number of rows), and n is the width of region o1 (i.e. the number of columns) It represents the value of the middle position (i, j) of the pixel area o1. If the pixel is located in the area o1, it is 1, otherwise it is 0. γ is the scale of the actual length of the oocyte image pixel.

[0070] S2, cytoplasmic area (um 2 ). The specific calculation formula is as follows:

[0071]

[0072] o2 is the oocyte egg region automatically segmented by the model, and the rest is the same as explained in S1.

[0073] S3, total internal area of ​​the inner ring of the transparent zone (um 2 ). The specific calculation formula is as follows:

[0074]

[0075] o3 is the inner region of the inner circle of the oocyte zona pellucida segmented automatically by the model, and the rest is the same as explained in S1.

[0076] S4, total internal area of ​​the outer ring of the transparent zone (um 2 ). The specific calculation formula is as follows:

[0077]

[0078] o4 is the inner region of the outer ring of the oocyte zona pellucida automatically segmented by the model, and the rest is the same as explained in S1.

[0079] S5, perivitelline space (um). The perivitelline space is defined as the length of the short axis of the vertical polar body. The specific calculation formula is as follows:

[0080]

[0081] in, Indicates the Y value of the upper boundary point corresponding to the x-coordinate point on the boundary of the polar region automatically segmented by the model. Corresponding to the Y value of the lower boundary point, d1 and d2 are the two pixels farthest from each other in the polar region O. is the x-coordinate value of the corresponding pixel point, and γ is the scale of the actual length of the oocyte image pixel.

[0082] S6. Oocyte diameter (um). The oocyte diameter is defined as the geometric mean of the long and short diameters of the oocyte. The specific calculation formula is as follows:

[0083]

[0084]

[0085]

[0086] Among them, d1 and d2 are the two pixels farthest from each other in the outer circle of the oocyte. The two pixels are perpendicular to d1d2 and intersect the outer circle of the oocyte. D1 is the length of the long diameter of the oocyte, and D2 is the length of the short diameter of the oocyte.

[0087] S7, transparent zone thickness (um). The transparent zone thickness is defined as the average value of the transparent zone thickness around the vertical coordinate system. The specific calculation formula is as follows:

[0088]

[0089] Among them, H belongs to the intersection of the vertical coordinate system and the outer circle of the oocyte, h belongs to the intersection of the vertical coordinate system and the inner circle of the oocyte, and x and y represent the coordinate system of the intersection.

[0090] See also Figure 4 The overall system architecture of the oocyte toxicity classification and evaluation method based on machine learning provided in the embodiment of the present application is described below. Figure 4 The overall architecture of the oocyte data analysis system shown in the figure consists of raw data input, and the feature extraction part consists of two parts: expert domain prior knowledge features and deep learning automatic extraction of black box features. The neural network is built end-to-end, and the parameters of the two branches are extracted separately but adjusted synchronously during training, that is, synchronous parameter adjustment in multiple feature fields.

[0091] Optionally, before inputting the oocyte image into the residual neural network model, the method also includes: preprocessing the original oocyte image to obtain the oocyte image; wherein the preprocessing process includes: increasing the brightness, contrast, boundary information and key point information of the original oocyte image based on a histogram equalization algorithm; normalizing the original oocyte image based on a set mean and a set variance; wherein the set mean is [0.485, 0.456, 0.406] and the set variance is [0.229, 0.224, 0.225].

[0092] Specifically, the preprocessing is mainly carried out from two aspects: using the histogram equalization method to increase the brightness contrast of the image, increase the boundary information and key point information, and use the image normalization parameters that are currently recognized in the field of computer vision, mean mean = [0.485, 0.456, 0.406], variance std = [0.229, 0.224, 0.225] to normalize the image. The specific operations are as follows, where image is the original pixel value of the image:

[0093] Nor=(Image / 255.0-mean) / std.

[0094] Optionally, the residual neural network model includes: a convolutional layer module and a residual module.

[0095] The convolutional layer module includes two 2D convolutional layers as adaptive image feature extractors. The shallow image features are extracted through the convolution process of the convolutional layer module. The model setting parameters use a kernel size of 3×3 and a stride of 2. Each convolutional layer is followed by a batch normalization layer and a linear correction unit.

[0096] Convolutional layer module, please refer to Figure 5. This embodiment uses two classic two-dimensional convolutional layers as adaptive image feature extractors to extract shallow image features through the convolution process. The model setting parameters use a kernel size of 3×3 and a step size of 2. Each convolutional layer is followed by a batch normalization layer and a linear rectification unit. The batch normalization layer is a trainable layer that aims to renormalize the data distribution and reduce the alternation caused by the convolutional layer. It helps to obtain more stable parameter updates during training. The linear rectification unit is a widely used activation function that is recognized for its ability to prevent the gradient vanishing problem in deep neural networks, and it zeroes the outputs of some neurons, which makes the network sparse and reduces the interdependence of parameters.

[0097] The residual module,includes three different types of 2D residual blocks, all with kernel size of 3×3 and stride of 2. The sizes of the convolution kernels are set to different sizes.

[0098] Residual module, please refer to Figure 6 . Three types of 2D residual blocks are used in this framework. Their kernel size is unified as 3×3 and the stride is 2, but the sizes of the convolution kernels are set to different sizes to capture more global unstructured features. It is worth noting that the original data before convolution in the residual block is input into the average pooling layer of the subsequent residual block. The average pooling layer converts samples into regional features and uses the average value of features in a region as the representative feature value to reduce the amount of calculation and the number of parameters. The two tensors of the same dimension after different processing are added to generate the final representation. The purpose is to inherit the optimization effect of the previous step and facilitate the convergence of the classification model.

[0099] Optionally, the adaptive attention network is an adaptive attention network for tabular data.

[0100] Specifically, since the features generated by deep learning and the features extracted by traditional methods come from different feature domains, have different value ranges and distributions, the traditional concatenation operation will ignore the meaning of the features and lack rationality and interpretability, and the order of feature arrangement will bring useless influence factors to the classification of the model. This invention uses an adaptive attention network for tabular data to map the two features into the same feature domain. The specific structure is as follows: Figure 7 shown.

[0101] Optionally, the method further includes: performing salient segment visualization on the features processed by the last residual block using back-propagation-based visualization methods Grad-CAM and GuidedGrad-CAM.

[0102] Specifically, in the present invention, the back-propagation-based visualization methods Grad-CAM and GuidedGrad-CAM are used to visualize the post-significant fragments of the features processed by the last residual module of the oocyte data. In addition, in order to distinguish the feature differences learned by the model based on the expert domain knowledge, the feature weights generated by the attention network are reversely mapped back to the input expert domain feature end to obtain the importance of the feature.

[0103] In summary, the embodiment of the present invention combines the advantages of deep learning and traditional feature engineering, which is neither limited to existing human experience nor completely dependent on simple data operations. The key features in the oocyte image are automatically extracted using deep learning, and the extracted features are screened in combination with the domain knowledge of experts. In addition, an attention mechanism is introduced so that the model can automatically adjust the weights of different features during the training process, highlight key information, and ignore noise data. This not only improves the performance of the model, but also makes the model more in line with biological principles, which helps us better understand and explain the classification process of oocytes. In this way, this method can not only give full play to the feature extraction ability of deep learning, but also ensure the interpretability and stability of the model.

[0104] The following is an explanation based on implementation cases:

[0105] The present invention has been tested respectively for the tasks in three stages:

[0106] (1) First, for the task of oocyte image segmentation, we conducted polar body, oocyte cytoplasm, zona pellucida outer ring, and zona pellucida inner ring segmentation experiments on 256 labeled samples, and evaluated the model based on some commonly used evaluation indicators. The experimental results are as follows Figure 8 shown.

[0107] (2) To solve the problem of automatic extraction of key features, the relevant features were calculated for 3402 oocytes in six groups, including PFOS, F-53B, non-poisoned, PFHxS, PFOA, and SCCP. The specific mean (± standard deviation) results are as follows: Fig. 9 shown.

[0108] (3) We set up two downstream tasks to verify the usability of the oocyte data analysis system that integrates deep learning methods and empirical knowledge: the first one is to determine whether the 16-hour mouse oocytes are contaminated. We verified it on a data set of 3402 images (including 2814 contaminated ones), and the model was able to achieve an accuracy of 88.12%. Secondly, we further classified and identified the specific contaminated subtypes, and the accuracy of the model was also able to reach 73.89%. The specific experimental results are shown in the figure. Fig.10 as well as Fig.11 shown.

[0109] Finally, in order to improve the visualization of the model interpretability, the present invention also provides several specific examples for the problem of whether the 16-hour mouse oocytes are contaminated. The importance distribution heat map of the image and the difference in feature importance can provide a more intuitive auxiliary effect on the analysis of oocytes, such as Fig.12 shown.

[0110] See also Fig.13 Based on the same inventive concept, the present application embodiment provides an oocyte toxicity classification and evaluation device based on machine learning, comprising:

[0111] An acquisition module 1301 is used to acquire an oocyte image; a processing module 1302 is used to process the oocyte image through a neural network segmentation model of a U-shaped structure to acquire key areas in the oocyte image; wherein the key areas include polar body areas, oocyte cytoplasm areas, oocyte outer ring areas and oocyte inner ring areas; a calculation module 1303 is used to calculate key features of the oocyte in the oocyte image based on the key areas and a mathematical constraint algorithm; wherein the key features are determined based on expert experience knowledge; an extraction module 1304 is used to extract local features of the oocyte in the oocyte image based on a residual neural network model; a mapping module 1305 is used to map the key features and the local features to the same feature domain based on an adaptive attention network; a determination module 1306 is used to determine the category of the oocyte in the oocyte image based on features of the same feature domain; wherein the category characterizes whether the oocyte is toxic.

[0112] It should be noted that the information interaction, execution process, etc. between the above-mentioned systems / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0113] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0114] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0115] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for classifying and evaluating oocyte toxicity based on machine learning, characterized in that: include: Acquire oocyte images; The oocyte image is processed by a U-shaped neural network segmentation model to obtain key areas in the oocyte image; wherein the key areas include a polar body area, an oocyte cytoplasm area, an oocyte outer ring area, and an oocyte inner ring area; Calculate the key features of the oocyte in the oocyte image based on the key area and the mathematical constraint algorithm; wherein the key features are automatically calculated and determined by constructing a mathematical formula based on expert experience knowledge; Extracting local features of the oocyte in the oocyte image based on a residual neural network model; Based on an adaptive attention network, mapping the key features and the local features to the same feature domain; Based on the features of the same feature domain, the category of the oocyte in the oocyte image is determined; wherein the category represents whether the oocyte is toxic.

2. The oocyte toxicity classification and evaluation method based on machine learning according to claim 1, characterized in that: The oocyte image is obtained by collecting images of mouse oocytes using an inverted phase contrast microscope; The oocyte image is stored in tif or tiff format, and the size of the oocyte image is 680*680.

3. The oocyte toxicity classification and evaluation method based on machine learning according to claim 1, characterized in that: The step of processing the oocyte image by using a U-shaped neural network segmentation model to obtain a key area in the oocyte image includes: Inputting the oocyte image into the neural network segmentation model of the U-shaped structure, and encoding the oocyte image through the encoding layer; Decoding the encoded image through a decoding layer to obtain a key area in the oocyte image; The low-level features acquired by the encoding layer are passed to the decoder at the same level in the decoding layer; the deep features decoded in the decoding layer are reversely passed to the encoder at the same level in the encoding layer; In the encoding layer, each convolution layer repeatedly uses two 3×3 unpadded convolution operations and is connected in series with the deep features from the decoding layer. Each convolution layer is connected to a linear rectification unit, a global maximum pooling operation of size 2×2, and a downsampling operation with an offset of 2; Among them, in the decoding layer, each layer is an iterative upsampling operation on the shallow features, each deconvolution layer performs a deconvolution operation of size 2×2, reduces the number of channels by half, and then connects it in series with the shallow features from the encoding layer, and then performs two convolution operations with a convolution kernel size of 3×3, and each deconvolution layer is followed by a linear correction unit; the last convolution layer adds a convolution operation of size 1×1 to map each component feature back to the size of the oocyte image.

4. The oocyte toxicity classification and evaluation method based on machine learning according to claim 3 is characterized in that: The loss function of the U-shaped neural network segmentation model is obtained by weighted calculation of the pixel-level softmax function and the cross entropy loss function.

5. The oocyte toxicity classification and evaluation method based on machine learning according to claim 1, characterized in that: The key features include at least the polar body area, the cytoplasm area, the total area inside the inner ring of the zona pellucida, the total area inside the outer ring of the zona pellucida, the perivitelline space, the oocyte diameter and the zona pellucida thickness.

6. The oocyte toxicity classification and evaluation method based on machine learning according to claim 1, characterized in that: Before inputting the oocyte image into the residual neural network model, the method further comprises: Preprocessing the original oocyte image to obtain the oocyte image; The pre-processing process includes: The brightness, contrast, boundary information and key point information of the original oocyte image are increased based on a histogram equalization algorithm; The original oocyte image is normalized based on a set mean and a set variance; wherein the set mean is [0.485, 0.456, 0.406], and the set variance is [0.229, 0.224, 0.225].

7. The oocyte toxicity classification and evaluation method based on machine learning according to claim 6, characterized in that: The residual neural network model includes: a convolutional layer module and a residual module; The convolutional layer module includes two two-dimensional convolutional layers as adaptive image feature extractors, and extracts shallow image features through the convolution process of the convolutional layer module. The model setting parameters use a kernel size of 3×3 and a step size of 2. Each convolutional layer is followed by a batch normalization layer and a linear correction unit; The residual module includes three different types of residual blocks, all of which have a kernel size of 3×3 and a step size of 2. The sizes of the convolution kernels are set to different sizes.

8. The oocyte toxicity classification and evaluation method based on machine learning according to claim 7, characterized in that: The method further comprises: The back-propagation based visualization methods Grad-CAM and Guided Grad-CAM visualize the salient fragments of the features processed by the last residual block.

9. The oocyte toxicity classification and evaluation method based on machine learning according to claim 8, characterized in that: The adaptive attention network is an adaptive attention network for tabular data.

10. An oocyte toxicity classification and evaluation device based on machine learning, characterized in that: include: An acquisition module, used for acquiring an oocyte image; A processing module is used to process the oocyte image through a U-shaped neural network segmentation model to obtain key areas in the oocyte image; wherein the key areas include a polar body area, an oocyte cytoplasm area, an oocyte outer ring area, and an oocyte inner ring area; A calculation module, used for calculating the key features of the oocyte in the oocyte image based on the key area and the mathematical constraint algorithm; wherein the key features are automatically calculated and determined by constructing a mathematical formula based on expert experience knowledge; An extraction module, used for extracting local features of the oocyte in the oocyte image based on a residual neural network model; A mapping module, used for mapping the key features and the local features to the same feature domain based on an adaptive attention network; The determination module is used to determine the category of the oocyte in the oocyte image based on the features of the same feature domain; wherein the category represents whether the oocyte is toxic.

Images