Method and related device for extracting follicle information based on ultrasound images

Abstract

This application relates to the field of ultrasound technology, and more particularly to a method and related device for extracting follicle information based on ultrasound images. The method addresses the problem of improving the accuracy of identifying multiple follicles in ultrasound images and enhancing the generalization of the follicle information extraction model. In this application, the ultrasound image is histogram-averaged to obtain a first image, which more clearly displays the edges of the follicles. The first image is input into an encoder module to obtain a first feature, which is then processed through a multi-scale multi-path fusion module to obtain a second feature. The second feature is then input into a shape flow module to obtain a third feature, further clarifying the follicle edges. The third feature and the second feature are then input into a decoder module to obtain a mask image. Based on the mask image, the number and / or size of follicles are extracted and displayed. This improves the accuracy of follicle identification, and the multi-scale multi-path fusion model enhances the ability to fuse high- and low-dimensional features, thereby improving the generalization of the follicle information extraction model.

Description

Technical Field

[0001] The present application relates to the field of ultrasound technology, and in particular to a method and related device for extracting follicle information based on ultrasound images. Background Art

[0002] When processing ultrasound images of follicles, the resulting images are blurry and noisy. Furthermore, the complex structure of the ovary, with follicles varying in size and number, makes it difficult to generalize the follicle recognition model. Ultrasound images also contain other hypoechoic areas besides follicles, such as blood vessels and cysts, which can easily lead to misidentification. While deep learning algorithms based on convolutional neural networks have improved segmentation accuracy in related technologies, they suffer from a high false positive rate when multiple, cluttered follicles are present. Therefore, improving the accuracy of multiple follicle recognition in ultrasound images and enhancing the generalization of follicle information extraction models are currently unresolved challenges. Summary of the Invention

[0003] The present application discloses a method and related device for extracting follicle information based on ultrasound images, which are used to solve the problem of how to improve the accuracy of identifying multiple follicles in ultrasound images and improve the generalization of the follicle information extraction model.

[0004] In a first aspect, the present application proposes a method for extracting follicle information based on ultrasound images, the method comprising:

[0005] Obtain ultrasound images of the ovarian follicles;

[0006] Performing histogram averaging on the ultrasound image to obtain a first image;

[0007] Performing a feature extraction operation on the first image using an encoder module to obtain a first feature;

[0008] Performing a feature extraction operation on the first feature using a multi-scale multi-channel fusion module to obtain a second feature;

[0009] performing a feature extraction operation on the second feature using a shape flow module to obtain a third feature;

[0010] Processing the second feature and the third feature using a decoder module to obtain a mask image of the follicle;

[0011] The number of follicles and / or size information of the follicles are extracted based on the mask image.

[0012] Optionally, the step of performing histogram averaging on the ultrasound image to obtain the first image specifically includes:

[0013] Obtaining grayscale values ​​of pixels in the ultrasound image;

[0014] The ultrasound image is mapped into a specified grayscale range based on the grayscale value of the ultrasound image to obtain the first image.

[0015] Optionally, the encoder module includes an even number of encoders, each encoder extracts a first sub-feature of a level, and the first feature includes the first sub-features of each level extracted by all encoders;

[0016] The multi-scale multi-path fusion module includes multiple dense block channels and a multi-level feature fusion module, each of the dense block channels includes multiple first dense block modules connected in series, the multi-level feature fusion module includes a second dense block module and a third dense block module, the number of feature paths output by the third dense block module is the same as the number of the encoders; the first dense block module, the second dense block module and the third dense block module each include two dense blocks, and the two dense blocks are used to exchange the feature information extracted by each.

[0017] The step of performing a feature extraction operation on the first feature using a multi-scale multi-channel fusion module to obtain a second feature specifically includes:

[0018] Input the first sub-features of two adjacent levels into the corresponding dense block channels to obtain the first intermediate features corresponding to the first sub-features of each level extracted by each dense block channel;

[0019] Using the second dense block module in the multi-level feature fusion module to process the first intermediate features corresponding to the first sub-features at different levels to obtain second intermediate features;

[0020] The second intermediate features, the first intermediate features at the lowest level, and the first intermediate features at the highest level are processed by the third dense block module to obtain second sub-features corresponding to each encoder, where the second features include each second sub-feature.

[0021] Optionally, the second feature includes n+1 second sub-features, where n+1 is an even number, and the shape flow module includes n groups of flow processing units, each group of flow processing units includes a residual module and a gated convolution module, wherein:

[0022] The input of the residual module of the first stream processing unit is the second sub-feature of the highest level, and the input of the gate convolution module is the output of the residual module of the first stream processing unit and the second sub-feature of the second highest level;

[0023] For each stream processing unit except the first stream processing unit, the input of the residual module of the stream processing unit is the output of the gate convolution module of the previous stream processing unit, and the input of the gate convolution module of the stream processing unit is the second sub-feature of the next level relative to the previous stream processing unit and the output of the residual module of the stream processing unit;

[0024] The output of the gated convolution module of the last stream processing unit is the third feature.

[0025] Optionally, the second feature includes n+1 second sub-features, the decoder module includes multiple decoders, the input of the first decoder is the output of the shape flow module and the second sub-feature of the highest level, the input of each decoder other than the first decoder is the output of the previous decoder and the second sub-feature of the level one level lower than the second sub-feature of the previous decoder, and the output of the last decoder is the mask image of the follicle.

[0026] Optionally, extracting the number of follicles and the size information of the follicles based on the mask image specifically includes:

[0027] Converting the mask image into a binary image;

[0028] Extracting the contours of each follicle based on the contour extraction method and obtaining the number of extracted follicles;

[0029] The size information of each follicle is obtained based on the outline of each follicle.

[0030] Optionally, the contour extraction method is used to extract the contours of each follicle and obtain the number of extracted follicles, specifically including:

[0031] Determine a closed area of ​​the binary image according to a watershed algorithm, and use points to outline the closed area in a specified order;

[0032] The number of the outlines was regarded as the number of extracted follicles.

[0033] Optionally, obtaining the size information of each follicle based on the outline of each follicle specifically includes:

[0034] Determine the longest first distance between any two points in the closed area according to the outline, and use the first distance as the major diameter;

[0035] In a direction perpendicular to the major diameter, determining the longest second distance between any two points in the closed area according to the outline, and taking the second distance as the minor diameter;

[0036] The contour, the major diameter, and the minor diameter are mapped on the ultrasound image, and the major diameter distance is marked.

[0037] In a second aspect, the present application proposes an ultrasound device, comprising:

[0038] a memory for storing processor-executable instructions;

[0039] A processor is configured to execute the instructions to implement any method provided in the first aspect of the present application.

[0040] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of the terminal device, enables the terminal device to execute any method provided in the first aspect of the present application.

[0041] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the methods provided in the first aspect of the present application.

[0042] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0043] In this application, an ultrasound image is processed through histogram averaging to obtain a first image, which makes the edges of the follicles clearer and improves the convergence speed of the neural network. The first image is input into the neural network's encoder module to obtain a first feature. The first feature is then input into a multi-scale multi-channel fusion module for feature fusion, thereby obtaining a second feature that reduces the impact of low-echo areas and improves the fusion capability of high- and low-dimensional features. The second feature is then input into a shape flow module to process the edge information of the follicles to obtain a third feature. The third feature and the second feature are output to a decoder module to obtain a mask image to be displayed. Finally, based on the mask image, information about the number and / or size of the follicles is extracted and displayed. Because the feature extraction results of the first feature at different levels are exchanged in the multi-scale multi-channel fusion module, the fusion capability of high- and low-dimensional features is improved. Therefore, even for ultrasound images with complex ovarian structures, a large number of follicles, and uneven edges, relatively accurate results can be obtained, thereby improving the generalization of the follicle information extraction model of this application.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A structural block diagram of the ultrasound device provided in an embodiment of the present application;

[0047] Figure 2a A schematic diagram of an ultrasound image provided in an embodiment of the present application;

[0048] Figure 2b A schematic diagram of an ultrasound image after histogram averaging provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the neural network module structure provided in an embodiment of the present application;

[0050] Figure 4a A schematic diagram of the structure of the encoder module provided in an embodiment of the present application;

[0051] Figure 4b A schematic diagram of the structure of the encoder provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the multi-scale multi-channel fusion module structure provided in an embodiment of the present application;

[0053] Figure 6a This is one of the schematic diagrams of the shape flow module structure provided in the embodiment of the present application;

[0054] Figure 6b This is the second structural diagram of the shape flow module provided in the embodiment of the present application;

[0055] Figure 7a A schematic diagram of the decoder module structure provided in an embodiment of the present application;

[0056] Figure 7b This is one of the schematic diagrams of the decoder structure provided in the embodiment of the present application;

[0057] Figure 7c The second schematic diagram of the decoder structure provided in the embodiment of the present application;

[0058] Figure 7d A schematic diagram of loss calculation provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the process of extracting follicle information based on ultrasound images provided in an embodiment of the present application;

[0060] Figure 9 A schematic diagram of the feature extraction process of the multi-scale multi-channel fusion module provided in an embodiment of the present application;

[0061] Figure 10 A schematic diagram of a binary image provided in an embodiment of the present application;

[0062] Figure 11 A schematic diagram of a process for extracting information on the number and / or size of follicles provided in an embodiment of the present application;

[0063] Figure 12 This is a schematic diagram of follicle information provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0065] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0066] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0067] Below, some terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0068] Hypoechoic areas: Different tissues have different densities, resulting in different degrees of sound wave attenuation, thus reflecting normal or abnormal tissue. The higher the echo, the greater the tissue density, and the darker the hypoechoic ultrasound image.

[0069] Dilated Convolution: The purpose of expanding the receptive field is achieved by filling the middle of the convolution kernel with 0 or sampling the input at equal intervals.

[0070] Receptive field: The size of the area on the ultrasound image mapped by the pixel points on the image output by the neural network.

[0071] Dropblock method: By setting the size and probability of the cropping area, a part of the adjacent whole area in the image is randomly cropped.

[0072] Mosaic algorithm: randomly crop the pictures in the same batch and then splice them into one picture as training data.

[0073] When processing ultrasound images of follicles, the resulting images are blurry and noisy, and the complex structure of the ovaries results in low accuracy and a narrow scope of application for traditional segmentation algorithms, making them only suitable for detecting a single, distinct mature follicle. Due to the varying size and number of follicles, the complex surrounding structure of the ovaries, and the presence of other hypoechoic areas besides the follicles, such as blood vessels and cysts, misidentification is prone to occur, making the follicle recognition model difficult to generalize. Although deep learning algorithms based on convolutional neural networks have improved segmentation accuracy in related technologies, they suffer from a high false positive rate when multiple, chaotically distributed follicles are present. Therefore, improving the accuracy of multiple follicle recognition in ultrasound images is a current challenge.

[0074] In view of this, an embodiment of the present application proposes a method and terminal device for extracting follicle information based on ultrasound images.

[0075] In this application, an ultrasound image is processed by histogram averaging to obtain a first image, making the edges of the follicles clearer and improving the convergence speed of the neural network. The first image is input into the encoder module of the neural network to obtain a first feature. The first feature is then input into the multi-scale multi-channel fusion module for feature fusion, which reduces the impact of low-echo areas and improves the fusion ability of high- and low-dimensional features to obtain a second feature. The second feature is then input into the shape flow module to process the edge information of the follicles to obtain a third feature. The third feature and the second feature are output to the decoder module to obtain a mask image to be displayed. Finally, based on the mask image, the number and / or size information of the follicles are extracted and displayed. Because the present application improves the fusion ability of high- and low-dimensional features, it can also obtain more accurate segmentation results for ultrasound images with complex ovarian structures, a large number of follicles, and jagged edges, thereby improving the generalization of the follicle information extraction model of the present application.

[0076] After introducing the design concepts of the embodiments of this application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of this application and are not limiting. In specific implementations, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0077] See also Figure 1 , which is a structural block diagram of the ultrasound device provided in an embodiment of the present application.

[0078] It should be understood that Figure 1 The ultrasound device 100 shown is only one example, and the ultrasound device 100 may have more Figure 1The more or less components shown in the figure can be combined with two or more components, or can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0079] Figure 1 Schematically shows a hardware configuration block diagram of the ultrasound apparatus 100 according to an exemplary embodiment.

[0080] like Figure 1 As shown, the ultrasound device 100 may include, for example: a processor 110, a memory 120, a display unit 130, and an ultrasound image acquisition device 140; wherein:

[0081] Ultrasonic image acquisition device 140, used for acquiring ultrasonic images;

[0082] A display unit 130 is used to display ultrasound images;

[0083] The memory 120 is configured to store data required for ultrasound images, which may include software programs, application interface data, etc.

[0084] The processor 110 is connected to the ultrasound image acquisition device 140 and the display unit 130 respectively, and is configured to execute the method for extracting follicle information based on ultrasound images.

[0085] To further illustrate the technical solutions provided by the embodiments of the present application, the following is a detailed description of the technical solutions in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative work. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiments of the present application.

[0086] This application includes three parts: image preprocessing, neural network model extraction of image features and follicle information extraction. In order to avoid the situation where the neural network model uses limited samples during training, which makes the model unable to be applied to ultrasound images in different situations, the following embodiments of this application explain the three parts of image preprocessing, neural network model extraction of image features and follicle information extraction respectively.

[0087] In the embodiment of the present application, after the ultrasonic image is acquired by the ultrasonic image acquisition device 140, the following steps are performed on the ultrasonic image in sequence.

[0088] 1. Preprocessing of Ultrasound Images

[0089] In the embodiments of this application, Figure 2aAs shown, there is noise in the obtained ultrasound image, and the brightness and darkness of different ultrasound images are different, and the brightness difference between the follicles and the background is not obvious. And because the brightness and darkness of each ultrasound image are different, the neural network will process the different brightness and darkness as features during training, which will reduce the speed of neural network convergence and even cause errors in the output results of the neural network. Therefore, in this application, after obtaining the ultrasound image, the image can be histogram-averaged to overcome or alleviate this problem. Histogram averaging can be implemented as follows: the grayscale values ​​of the pixels in the ultrasound image are collected to form a grayscale histogram. The grayscale in the grayscale histogram is then stretched or compressed to a specified grayscale range to obtain a first image. As shown Figure 2b As shown in the figure, after histogram averaging, the brightness and contrast of the ultrasound image are improved, the edge of the follicle is more clearly displayed, and the image is unified into a specified grayscale range, which can improve the convergence speed of the neural network.

[0090] 2. Neural Network Model Extraction of Image Features

[0091] In the embodiment of the present application, the edge of the follicle in the ultrasound image is not clear and the same image often contains multiple follicles. In order to accurately extract the characteristics of multiple follicles, the present application designs the following Figure 3 The neural network model shown includes an encoder module, a multi-scale multi-path fusion module, a shape flow module and a decoder module, and also provides a way to optimize the model by adjusting the model using a loss function.

[0092] 1) Structure of the neural network model

[0093] ①Encoder module

[0094] like Figure 4a , which is a schematic structural diagram of an encoder module. The encoder module includes an even number of encoders. Each encoder extracts the first sub-features of a level. The first features include the first sub-features of each level extracted by all encoders.

[0095] The structure of each encoder is as follows Figure 4b As shown. Figure 4b In the example, the input of each encoder is the first sub-feature output by the previous encoder, and the output of each encoder is the first sub-feature, where the input of the first encoder is the first image. Each encoder passes the input data through a 3*3 convolution layer, a normalization + ReLU function layer, a 3*3 convolution layer, a normalization + ReLU function layer, an attention layer, and a maximum pooling layer, and then outputs the first sub-feature.

[0096] ②Multi-scale multi-channel fusion module

[0097] like Figure 5As shown, a structure of a multi-scale multi-channel fusion module including three dense block modules is exemplarily given, and the number of dense blocks in each dense block module is an even number.

[0098] In the embodiment of the present application, the multi-scale multi-path fusion module includes multiple dense block channels and a multi-level feature fusion module, each dense block channel includes multiple first dense block modules connected in series, and the multi-level feature fusion module includes a second dense block module and a third dense block module respectively. The number of feature paths output by the third dense block module is the same as the number of encoders; the first dense block module, the second dense block module and the third dense block module include at least two dense blocks, and the two dense blocks are used to exchange the feature information extracted by each. Figure 5 The specific structure of the multi-scale multi-channel fusion module is given as an example after the first feature composed of four layers of first sub-features is input into the multi-scale multi-channel fusion module.

[0099] exist Figure 5 In the figure, the dotted line indicates the operation of two dense blocks exchanging their extracted feature information, and the solid line indicates the input of the data of a dense block to the next dense block. For example, the first dense block module includes eight dense blocks, the second dense block module includes two dense blocks, and the third dense block module includes four dense blocks, where every two dense blocks exchange their extracted feature information twice. If the encoder module contains four encoders, then in the first dense block module, the four dense blocks in the first column are input from top to bottom to the first sub-feature output by encoder 1, the first sub-feature output by encoder 2, the first sub-feature output by encoder 3, and the first sub-feature output by encoder 4.

[0100] In the four dense blocks in the first column, the extracted feature information is exchanged twice between the first and second dense blocks from top to bottom, and the extracted feature information is exchanged twice between the third and fourth dense blocks from top to bottom. The output results of the first column of dense blocks are then input into the four dense blocks in the second column of the first dense block module in order from top to bottom, and the same operation of exchanging the extracted feature information twice is performed in the four dense blocks in the second column as in the four dense blocks in the first column to obtain the first intermediate feature, where the first intermediate feature includes four first intermediate sub-features. The first intermediate feature of the lowest level is input into the first dense block from top to bottom in the third dense block module, the first intermediate feature of the highest level is input into the last dense block from top to bottom in the third dense block module, and the remaining first sub-features are input into the dense blocks in the second dense block module in order from top to bottom.

[0101] In the second dense block module, the first intermediate features at different levels are extracted between the two dense blocks and then the extracted feature information is exchanged twice to obtain the second intermediate features, where the second intermediate features include multiple layers of sub-features, and the number of these sub-features is two less than the number of the first sub-features. The second intermediate features are sequentially input to the dense blocks in the third dense block module from top to bottom.

[0102] In the third dense block module, after the second intermediate features, the first intermediate features of the lowest level, and the first intermediate features of the highest level are extracted in the dense block, the extracted feature information is exchanged twice between every two dense blocks to output the second feature, where the second feature includes multiple layers of second sub-features, and the number of the second sub-features is the same as the number of the first sub-features.

[0103] Furthermore, in order to enhance the ability of the multi-scale multi-channel fusion module to fuse the first features at different levels, Figure 5 In the structure shown, an even number of dense blocks can be added to the first dense block module, the second dense block module, and the third dense block module, or the number of times each two dense blocks exchange their extracted feature information can be increased. This application does not impose any restrictions on this.

[0104] ③Shape flow module

[0105] In order to make the edge of the follicle clearer, the present application also provides the second features at different levels. Figure 6a The stream processing units in the shape stream module shown perform feature extraction. Each stream processing unit includes a residual module and a gated convolution module. The input to the gated convolution module is the output of the residual module and a second feature one level lower than the input of the residual module. The output of the last gated convolution module is used as the third feature.

[0106] In the embodiment of the present application, taking the second sub-feature of the four-layer as an example, the structure of the shape flow module is as follows: Figure 6a As shown. Figure 6a In the figure, there are three stream processing units. Each stream processing unit includes a residual module and a gated convolution module. The input of the first layer second sub-feature is the second feature of the lowest level, and the fourth layer second sub-feature is the second feature of the highest level. From left to right, the input of the residual module of the first stream processing unit is the fourth layer second sub-feature. The output of the residual module and the third layer second sub-feature are input to the gated convolution module.

[0107] For each stream processing unit except the first one, the input of the residual module of the stream processing unit is the output of the gated convolution module of the previous stream processing unit, and the input of the gated convolution module of the stream processing unit is the output of the fully connected layer. The output of the gated convolution module of the last stream processing unit is the third feature.

[0108] exist Figure 6a In the example, if the input second feature includes n+1 second sub-features and n+1 is an even number, the shape flow module includes n groups of flow processing units.

[0109] In another embodiment of the present application, in order to further improve the accuracy of feature extraction by the shape flow module, as shown in FIG. Figure 6b As shown, a fully connected layer is added to each stream processing unit, and the output of the residual module is concatenated with the second sub-feature of the third layer in the fully connected layer and then input into the gate convolution module.

[0110] ④Decoder module

[0111] like Figure 7a As shown in the figure, the decoder module includes multiple decoders. The input of each decoder is the second feature of a different level and the output of the decoder of the previous level. The input of the top decoder is the third feature and the second feature of the highest level. In the decoder module, the features of different levels are mapped to the mask image.

[0112] In the decoder module, the decoders are arranged in order from high to low. Except for the top-level decoder, the input of each decoder is the second sub-feature and the output of the previous decoder. The input of the top-level decoder is the third feature and the second feature of the highest layer, and the bottom-level decoder outputs the mask image.

[0113] The structure of each decoder is as follows Figure 7b As shown in the figure, the input 1 of the top decoder is the third feature, the input 1 of the remaining decoders is the output of the previous decoder, and the input 2 of the decoder is the multi-layer second feature output by the multi-scale multi-channel fusion module. After the input 1 is subjected to feature extraction by the deconvolution layer 1, it is concatenated with the input 2 in the fully connected layer 1. The concatenated result is sequentially input into the 3*3 convolution layer 1, the 1*1 convolution layer 1, and the Sigomid function layer 1. The result of the 3*3 convolution layer 1 is then input into the attention layer to extract the attention feature. The attention feature is then multiplied by the result of the 3*3 convolution layer 1, and then multiplied by the output of the Sigomid function layer 1 to obtain the first product.

[0114] Input 2 is passed through 1*1 convolution layer 2 and added to input 1. The sum is passed through Relu function layer, 1*1 convolution layer 3, Sigomid function layer 2 and deconvolution layer 2 in sequence for feature extraction. The result of feature extraction is multiplied by input 2 to obtain the second product, and the second product is input into fully connected layer 3.

[0115] Input 1 passes through deconvolution layer 3 and is then fed into fully connected layer 3, where it is concatenated with the second product. The output of fully connected layer 3 is concatenated with the first product in fully connected layer 2, and the concatenated result is fed into 1*1 convolution layer 4. This is then fed into 3*3 convolution layer 2, normalization + Relu function layer 1, 3*3 convolution layer 3, and normalization + Relu function layer 2 in sequence. The output of normalization + Relu function layer 2 is added to the output of 1*1 convolution layer 4, and the result is fed into the next decoder.

[0116] In addition, in order to better extract the feature information of the image, the present application can also Figure 7b The deconvolution layer in is replaced by Figure 7c The dilated convolution in the image is to extract the features of input 1 through the dilated convolution layer 1, and then concatenate it with input 2 in the fully connected layer 1. The concatenated result is sequentially input into the 3*3 convolution layer 1, the 1*1 convolution layer 1 and the Sigomid function layer 1. The result of the 3*3 convolution layer 1 is then input into the attention layer to extract the attention features. The attention features are multiplied by the results of the 3*3 convolution layer 1, and then multiplied by the output results of the Sigomid function layer 1 to obtain the first product.

[0117] Input 2 is passed through 1*1 convolution layer 2 and added to input 1. The sum is passed through Relu function layer, 1*1 convolution layer 3, Sigomid function layer 2 and hole convolution layer 2 in sequence for feature extraction. The result of feature extraction is multiplied by input 2 to obtain the second product, and the second product is input into fully connected layer 3.

[0118] Input 1 passes through the dilated convolution layer 3 and is then input into the fully connected layer 3, where it is concatenated with the second product. The output of the fully connected layer 3 is concatenated with the first product in the fully connected layer 2, and the concatenated result is input into the 1*1 convolution layer 4. This is then sequentially input into the 3*3 convolution layer 2, the normalization + Relu function layer 1, the 3*3 convolution layer 3, and the normalization + Relu function layer 2. The output of the normalization + Relu function layer 2 is added to the output of the 1*1 convolution layer 4, and the result of the addition is output to the next decoder.

[0119] Therefore, the receptive field can be increased by replacing the deconvolution layer with a dilated convolution layer.

[0120] 2) Training of neural network models

[0121] In the embodiment of the present application, since the collected ultrasound images are not exactly the same, the neural network model needs to be trained with a large amount of data before a more accurate feature extraction result of the ultrasound image can be obtained. Therefore, in the embodiment of the present application, the follicles in the ultrasound image that are larger than the preset diameter are used as the preset standard results, that is, the follicles that need to be detected, and the edges of such follicles are marked. The output results of the decoders other than the top decoder are compared with the preset standard results to obtain the following: Figure 7d The loss is calculated and weights are assigned to the losses corresponding to different decoders based on its own experience. The loss function is obtained by weighted summation of all differences according to the weights. Finally, the parameters in the neural network model are adjusted according to the loss function to make the output of the neural network model more accurate.

[0122] 3. Follicle Information Extraction

[0123] This application converts the mask image that cannot be displayed directly into a binary image, and then uses the watershed algorithm to calculate the contour, long diameter, short diameter and long diameter distance of the follicle, and marks the contour, long diameter, short diameter and long diameter distance at the corresponding positions of the ultrasound image.

[0124] After summarizing the method for extracting follicle information based on ultrasound images of the present application, the method for extracting follicle information based on ultrasound images of the present application will be further introduced with reference to diagrams.

[0125] like Figure 8 The figure shows a flow chart of the method for extracting follicle information based on ultrasound images provided in an embodiment of the present application.

[0126] In step 201 , an ultrasound image of an ovarian follicle is acquired.

[0127] In step 202, the ultrasound image is histogram-averaged to obtain a first image.

[0128] In step 203, an encoder module is used to perform a feature extraction operation on the first image to obtain a first feature.

[0129] In step 204, a multi-scale multi-channel fusion module is used to perform a feature extraction operation on the first feature to obtain a second feature.

[0130] After the first feature is input into the multi-scale multi-channel fusion module, Figure 9 The steps shown perform the following operations.

[0131] In step 301, the first sub-features of two adjacent levels are input into corresponding dense block channels to obtain first intermediate features corresponding to the first sub-features of each level extracted by each dense block channel.

[0132] In the embodiments of this application, Figure 5As shown in the figure, the first sub-feature output by encoder 1 is input to dense block 1, the first sub-feature output by encoder 2 is input to dense block 2, the first sub-feature output by encoder 3 is input to dense block 5, and the first sub-feature output by encoder 4 is input to dense block 6. After that, the extracted features of dense blocks 1 and 2 are exchanged with those of dense blocks 5 and 6, and the feature extraction operation is performed again. After the two exchanges, the result of dense block 1 is input to dense block 3, the result of dense block 2 is input to dense block 4, the result of dense block 5 is input to dense block 7, and the result of dense block 6 is input to dense block 8.

[0133] Dense blocks 3, 4 and dense blocks 7, 8 exchange their extracted features and perform feature extraction operations again. After exchanging twice, they output the first intermediate features.

[0134] In step 302, the second dense block module in the multi-level feature fusion module is used to process the first intermediate features corresponding to the first sub-features at different levels to obtain second intermediate features.

[0135] In the embodiment of the present application, the first intermediate feature corresponding to dense block 4 is input to dense block 9 of the second dense block module, and the first intermediate feature corresponding to dense block 7 is input to dense block 10 of the second dense block module.

[0136] The dense blocks 9 and 10 exchange their respective extracted features and perform feature extraction operations again. After exchanging twice, the second intermediate features output by the dense blocks 9 and 10 are obtained.

[0137] In step 303, the second intermediate features, the first intermediate features at the lowest level, and the first intermediate features at the highest level are processed by a third dense block module to obtain second sub-features corresponding to each encoder.

[0138] In the embodiment of the present application, Figure 5 The first intermediate feature corresponding to dense block 3 is input to dense block 11 of the third dense block module, the first intermediate feature corresponding to dense block 8 is input to dense block 14 of the third dense block module, the second intermediate feature corresponding to dense block 9 is input to dense block 12 of the third dense block module, and the second intermediate feature corresponding to dense block 10 is input to dense block 13 of the third dense block module. The features extracted by dense blocks 11 and 12 and dense blocks 13 and 14 are then swapped and feature extraction is performed again. After swapping twice, the second feature is output. The second feature includes multiple layers of second sub-features.

[0139] In the present application, the dense block in the multi-scale multi-path fusion module extracts the first feature and exchanges the feature information extracted by each of the two dense blocks multiple times, which can better fuse the first sub-features at different levels and improve the fusion capability of high- and low-dimensional features.

[0140] In step 205, a shape flow module is used to perform a feature extraction operation on the second feature to obtain a third feature.

[0141] In step 206, the second feature and the third feature are processed by a decoder module to obtain a mask image of the follicle.

[0142] In order to better understand the relationship between the encoder module, decoder module, multi-scale multi-channel fusion module and shape flow module, this application is based on Figure 3 The feature extraction process of the first image by the neural network is further explained.

[0143] The first image is extracted through an even number of encoders in the encoder module to obtain the first feature, which includes first sub-features at different levels. The first feature is input into the multi-scale multi-channel fusion module. After feature extraction, the second feature is obtained. The second feature includes second sub-features at different levels, and the number of levels of the second sub-feature is the same as the number of levels of the first sub-feature. The second sub-feature is input into the shape flow module to process the edge of the follicle to obtain the third feature. The third feature and the second feature are input into the decoder module, and the features of each level are mapped to the image to obtain a mask image. Figure 3 In the above figure, the hierarchical order of the encoder module output is the same as the hierarchical order of the multi-scale multi-channel fusion module output and has a corresponding relationship, that is, the second sub-feature of the first layer output by the multi-scale multi-channel fusion module corresponds to the first sub-feature of the first layer output by the encoder module.

[0144] In step 207 , the number of follicles and / or the size information of the follicles are extracted based on the mask image.

[0145] In the embodiment of the present application, since the mask image cannot be displayed directly, before extracting the outline of each follicle to obtain the size information of each follicle, the mask image needs to be converted into the following image: Figure 10 The binary image shown is used, and the boundaries and number of follicles in the binary image are read through the watershed algorithm. In the watershed algorithm, the gradient of the grayscale value of the pixel points in the image is calculated. Since the grayscale value of the follicle in the binary image is 1 and the grayscale value of the rest of the area is 0, the gradient change at the follicle boundary is the most obvious. At this time, the watershed algorithm uses the place where the gradient change is the most obvious as the boundary of the follicle, and the connected boundary as a closed area. The number of closed areas is output as the number of follicles, and the boundaries of the closed areas are marked out with points in a clockwise or counterclockwise manner to obtain the outline of each follicle. Figure 10 In order to distinguish the spots from the follicles and background areas, the spots are displayed in gray.

[0146] The number of follicles and the outline of each follicle are determined by the watershed algorithm, and then the Figure 11 The steps shown obtain the size information of each follicle based on the outline of each follicle on the binary image, and mark the size information of each follicle on the ultrasound image, wherein the size information of each follicle includes the follicle outline, long diameter, short diameter and long diameter distance.

[0147] In step 401, the longest first distance between any two points in the closed area is determined according to the outline, and the first distance is used as the major diameter.

[0148] In step 402, in a direction perpendicular to the long diameter distance, the longest second distance between any two points in the closed area is determined according to the contour, and the second distance is used as the short diameter.

[0149] In step 403, the contour, major diameter, and minor diameter are mapped on the ultrasound image, and the major diameter distance is marked.

[0150] In the embodiment of the present application, since the final display is an ultrasound image, the size information of the follicles needs to be marked on the Figure 12 The ultrasound image shown allows the user to directly observe the shape and size of the follicles.

[0151] In another embodiment of the present application, to allow users to more intuitively observe the size of the follicle, after determining the follicle outline in the binary image, the present application establishes a two-dimensional coordinate system and obtains the coordinates of each point on the follicle outline. The minimum horizontal coordinate of each point in the follicle outline is used as the horizontal coordinate of the two left vertices of the circumscribed rectangle, the maximum horizontal coordinate of the point is used as the horizontal coordinate of the two right vertices of the circumscribed rectangle, the minimum vertical coordinate of the point is used as the vertical coordinate of the two lower vertices of the circumscribed rectangle, and the maximum vertical coordinate of the point is used as the vertical coordinate of the two upper vertices of the circumscribed rectangle. The horizontal and vertical coordinates of the four vertices are used to determine the follicle's horizontal or vertical circumscribed rectangle, and the circumscribed rectangle is mapped to the corresponding position on the ultrasound image.

[0152] Therefore, after eliminating the noise of the ultrasound image through histogram averaging and increasing the convergence speed of the neural network module, the present application fuses the high- and low-dimensional features of the first image based on the neural network module and obtains a clearer follicle boundary. Then, the size information of the follicle is determined based on the mask image output by the neural network and mapped to the corresponding position of the ultrasound image, so that the user can directly observe the number and size of the follicles on the ultrasound image. The multi-scale multi-path fusion module and the shape flow module can simultaneously process high- and low-dimensional features and make the follicle boundary smoother. Therefore, when the network model of the present application processes ultrasound images with a large number of follicles, uneven follicle edges, and complex ovarian structures, it can obtain more accurate results, thereby improving the generalization of the follicle information extraction model of the present application.

[0153] In one embodiment of the present application, to train the neural network model's ability to capture detail, after histogram averaging, the present application uses the Dropblock method to set the size of the region to be cropped and the Bernoulli function probability of cropping regions of the same size at different locations in the ultrasound image. The number of regions to be cropped is controlled by the Bernoulli probability. To ensure that the region to be cropped is within the ultrasound image, a valid region is set within the ultrasound image to ensure that the cropped region is within the valid region. After determining the valid region, the region to be cropped is determined by multiplying the Bernoulli function probability, the size of the valid region, and the region size. The present application's method for extracting follicle information from ultrasound images is then executed on the cropped ultrasound image. Because some details in the ultrasound image are cropped by the Dropblock method, the neural network model needs to extract more details to compensate for the cropped details. Therefore, by combining the Dropblock method with the present application's method for extracting follicle information from ultrasound images multiple times, the present application's ability to capture ultrasound image details, improves the accuracy of follicle information extraction, and enhances the generalization of the present application's method.

[0154] In another embodiment of the present application, in order to train the neural network model's ability to capture details, after histogram averaging, a mosaic algorithm can be used to set the region center and region size in the ultrasound image, and replace the pixel values ​​of the pixels in the region with random pixel values ​​to improve the generalization of the model. There are many ways to improve the generalization of the method of the present application, including but not limited to the method described in the present application, which is not limited to this application.

[0155] In an exemplary embodiment, the present application further provides a computer-readable storage medium including instructions, such as a memory 120 including instructions. The instructions can be executed by the processor 110 of the ultrasound device 100 to implement the above-mentioned ultrasound image-based follicle information extraction method. Alternatively, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0156] In an exemplary embodiment, a computer program product is further provided, including a computer program. When the computer program is executed by the processor 110, the method for extracting follicle information based on ultrasound images as provided in the present application is implemented.

[0157] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for extracting follicle information based on ultrasound images, characterized in that: The method comprises: Obtain ultrasound images of the ovarian follicles; Performing histogram averaging on the ultrasound image to obtain a first image; Performing a feature extraction operation on the first image using an encoder module to obtain a first feature; Performing a feature extraction operation on the first feature using a multi-scale multi-channel fusion module to obtain a second feature; Performing a feature extraction operation on the second feature using a shape flow module to obtain a third feature; Processing the second feature and the third feature using a decoder module to obtain a mask image of the follicle; extracting the number of follicles and / or size information of the follicles based on the mask image; The encoder module includes an even number of encoders, each encoder extracts a first sub-feature of a level, and the first feature includes the first sub-features of each level extracted by all encoders; The multi-scale multi-path fusion module includes multiple dense block channels and a multi-level feature fusion module, each of the dense block channels includes multiple first dense block modules connected in series, the multi-level feature fusion module includes a second dense block module and a third dense block module, the number of feature paths output by the third dense block module is the same as the number of the encoders; the first dense block module, the second dense block module and the third dense block module each include two dense blocks, and the two dense blocks are used to exchange the feature information extracted by each. The step of performing a feature extraction operation on the first feature using a multi-scale multi-channel fusion module to obtain a second feature specifically includes: Input the first sub-features of two adjacent levels into the corresponding dense block channels to obtain the first intermediate features corresponding to the first sub-features of each level extracted by each dense block channel; Using the second dense block module in the multi-level feature fusion module to process the first intermediate features corresponding to the first sub-features at different levels to obtain second intermediate features; The second intermediate features, the first intermediate features at the lowest level, and the first intermediate features at the highest level are processed by the third dense block module to obtain second sub-features corresponding to each encoder, where the second features include each second sub-feature.

2. The method according to claim 1, characterized in that The step of performing histogram averaging on the ultrasound image to obtain the first image specifically includes: Obtaining grayscale values ​​of pixels in the ultrasound image; The ultrasound image is mapped into a specified grayscale range based on the grayscale value of the ultrasound image to obtain the first image.

3. The method according to claim 1, characterized in that The second feature includes n+1 second sub-features, where n+1 is an even number. The shape flow module includes n groups of flow processing units, each group of flow processing units includes a residual module and a gated convolution module, wherein: The input of the residual module of the first stream processing unit is the second sub-feature of the highest level, and the input of the gate convolution module is the output of the residual module of the first stream processing unit and the second sub-feature of the second highest level; For each stream processing unit except the first stream processing unit, the input of the residual module of the stream processing unit is the output of the gate convolution module of the previous stream processing unit, and the input of the gate convolution module of the stream processing unit is the second sub-feature of the next level relative to the previous stream processing unit and the output of the residual module of the stream processing unit; The output of the gated convolution module of the last stream processing unit is the third feature.

4. The method according to claim 1, wherein The second feature includes n+1 second sub-features, the decoder module includes multiple decoders, the input of the first decoder is the output of the shape flow module and the second sub-feature of the highest level, the input of each decoder other than the first decoder is the output of the previous decoder and the second sub-feature of the level one level lower than the second sub-feature of the previous decoder, and the output of the last decoder is the mask image of the follicle.

5. The method according to claim 1, wherein The step of extracting the number of follicles and the size information of the follicles based on the mask image specifically includes: Converting the mask image into a binary image; Extracting the contours of each follicle based on the contour extraction method and obtaining the number of extracted follicles; The size information of each follicle is obtained based on the outline of each follicle.

6. The method according to claim 5, characterized in that The method of extracting the contour of each follicle and obtaining the number of extracted follicles specifically includes: Determine a closed area of ​​the binary image according to a watershed algorithm, and use points to outline the closed area in a specified order; The number of the outlines was regarded as the number of extracted follicles.

7. The method according to claim 6, characterized in that The step of obtaining the size information of each follicle based on the outline of each follicle specifically includes: Determine the longest first distance between any two points in the closed area according to the outline, and use the first distance as the major diameter; In a direction perpendicular to the major diameter, determining the longest second distance between any two points in the closed area according to the outline, and taking the second distance as the minor diameter; The contour, the major diameter, and the minor diameter are mapped on the ultrasound image, and the major diameter distance is marked.

8. An ultrasonic device, characterized in that include: a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement any one of the methods provided in claims 1-7 of the present application.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an ultrasound device, the ultrasound device is enabled to perform any one of the methods provided in claims 1 to 7 of the present application.

Images