Liver lesion image description method based on semantic segmentation network

Abstract

The application discloses a liver lesion image description method based on a semantic segmentation network, comprising the following steps: building a Unet semantic segmentation network based on a lightweight network GhostNet, and constructing a liver lesion segmentation model; acquiring a historical liver lesion ultrasound image dataset, inputting the liver lesion segmentation model for training, and obtaining an optimal segmentation model; and describing the features of the lesion image based on the segmentation result. The application segments the liver lesion by constructing a lesion segmentation model, and describes based on the segmentation result, thereby improving the accuracy and reliability of liver disease diagnosis, reducing misjudgment and subjective bias; by combining the advantages of deep learning and traditional image processing, the characteristics of both are fully utilized, the performance and stability of the algorithm are improved; automatic processing and analysis of the ultrasound image are realized, the lesion description result provides accurate disease positioning and type judgment, and a better treatment scheme is provided for doctors, thereby improving the treatment effect and treatment experience of patients.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, and in particular to a liver lesion image description method based on a semantic segmentation network. BACKGROUND

[0002] Early diagnosis of liver diseases is very important for the treatment and recovery of patients. Traditional liver image diagnosis methods mainly rely on the experience and manual analysis of doctors, which has the problems of subjectivity, long time consumption and being easily affected by the level of operators. In recent years, with the development of deep learning, intelligent recognition algorithms based on images have made significant progress in the field of medical imaging.

[0003] Liver lesion segmentation and description is an important medical image analysis technology based on ultrasound images, and plays a key role in liver disease diagnosis. Ultrasound imaging is a non-invasive, non-radiation and low-cost imaging technology, and has become one of the common examination methods for liver diseases.

[0004] Ultrasound imaging generates real-time images of the liver by using the principle of propagation and reflection of high-frequency sound waves in human tissues. It can provide direct observation of the internal structure and abnormal lesions of the liver, and has good safety and repeatability. However, the characteristics of liver ultrasound images also bring some challenges to lesion segmentation and description. Ultrasound images often have low contrast, noise interference and image blur, which makes it difficult to distinguish liver lesions from surrounding tissues. In addition, different types of liver lesions have different manifestations in ultrasound images, such as differences in shape, boundary blur, internal echo characteristics, etc., which increase the difficulty of accurate segmentation and description. This leads to the need for liver lesions to be measured and described accurately and quickly by clinicians with high levels of clinical experience and in a relatively focused state. With the help of deep learning, image processing and other technologies, doctors can assist in completing these tasks, automatically fill in lesion size and description information, and allow doctors to devote their time and energy to more valuable diagnosis and treatment, while improving the patient's experience of visiting the doctor. SUMMARY

[0005] The purpose of the present application is to provide a liver lesion image description method based on a semantic segmentation network to solve the problem of inaccurate lesion analysis and long diagnosis time caused by subjective factors in the prior art, which affects the patient's experience of visiting the doctor.

[0006] To achieve one of the above-mentioned purposes, an embodiment of the present application provides a liver lesion image description method based on a semantic segmentation network, which comprises:

[0007] An Unet semantic segmentation network is built based on a lightweight network GhostNet, and a liver lesion segmentation model is constructed;

[0008] Obtain a historical liver lesion ultrasound image dataset, input the liver lesion segmentation model for training, and obtain an optimal segmentation model;

[0009] Based on the segmentation result, the characteristics of the lesion image are described.

[0010] As a further improvement of an embodiment of the application, the method further comprises: the Unet semantic segmentation network is composed of a symmetrical down-sampling path encoder and an up-sampling path decoder;

[0011] The Ghost module based on the lightweight network GhostNet is used to build the convolution unit in the encoder and the decoder of the Unet semantic segmentation network;

[0012] The Ghost module is used to extract more image features, including,

[0013] The input information X is defined as R c×h×w , c is the input channel, h and w are the height and width respectively, and R is the real number field;

[0014] The input information is passed through a convolution kernel f' with m filters and a kernel size of k*x to obtain m output feature maps Y', and the output feature map Y' is represented as:

[0015] Y'=X*f'

[0016] The m output feature maps Y' are transformed linearly to obtain n output feature maps Y, where m*s=n, n>=m, and Y is the output feature of the Ghost module;

[0017] The formula of the linear transformation is:

[0018]

[0019] Where, y i ′ is the i-th feature map in Y', and Φ i,j (y i ′) represents the j-th linear transformation on the i-th feature map to generate the j-th feature map y ij .

[0020] As a further improvement of an embodiment of the application, the method further comprises: constructing a Ghost bottleneck unit based on the Ghost module;

[0021] The Ghost bottleneck unit is composed of a first Ghost module, an attention module and a second Ghost module in series, and there is a jump link from the input end to the output end;

[0022] The first Ghost module is used to expand the number of channels, the attention module is used to strengthen the feature extraction capability of the Ghost module, and the second Ghost module is used to reduce the number of channels;

[0023] Batch normalization and a Hard-swish activation function are applied after the first Ghost module, and batch normalization is applied after the second Ghost module;

[0024] The activation function is represented as:

[0025]

[0026] As a further improvement of an embodiment of the application, the method further comprises: inputting the lesion image into the liver lesion segmentation model at different scales respectively, extracting a feature image by an encoder corresponding to the scale, and then sequentially passing through a feature image decoder of the corresponding scale encoder to obtain a final output result;

[0027] The scales of the encoder and the decoder include 1, 1 / 2, 1 / 4, 1 / 8 and 1 / 16;

[0028] The encoder is composed of 2 Ghost bottleneck units to form a stage, and there are 5 stages in total, and the feature map is down-sampled by 2 times per stage;

[0029] The decoder is composed of 2 Ghost bottleneck units to form a stage, and there are 5 stages in total, and the feature map is up-sampled by 2 times per stage.

[0030] As a further improvement of an embodiment of the application, the method further comprises: adding an atrous spatial convolution pooling layer ASPP to the input layer of the liver lesion segmentation model;

[0031] Among them, the feature image with a coding scale of 1 / 16 needs to pass through the atrous spatial convolution pooling layer ASPP before passing through the decoder, and the feature images of other scales are subjected to a channel dimension splicing operation;

[0032] The atrous spatial convolution pooling layer ASPP includes 4 parallel atrous convolutions and 1 global average pooling;

[0033] The 4 parallel atrous convolutions include a 1*1 convolution, a 3*3 convolution with an expansion coefficient of 3, a 3*3 convolution with an expansion coefficient of 6, and a 3*3 convolution with an expansion coefficient of 9;

[0034] The global average pooling obtains up-sampling through a 1*1 convolution operation;

[0035] The four parallel hollow convolution feature images and the up-sampling splicing are spliced together, and a final output is obtained through a 1*1 convolution operation.

[0036] The final output includes different levels of receptive fields.

[0037] As a further improvement of an embodiment of the application, the method further comprises training the liver lesion segmentation model, including,

[0038] The Dice-Loss loss function and the cross-entropy loss function are used for the output scale of 1, and the cross-entropy loss function is used for other scale output results.

[0039] The Dice-Loss loss function is represented as:

[0040]

[0041] Wherein, X represents the pixel label of the real segmentation image, Y represents the pixel category of the model predicted segmentation image, |X| and |Y| represent the number of elements in X and Y respectively, and |X∩Y| represents the number of intersection elements between X and Y.

[0042] The cross-entropy loss function is represented as:

[0043]

[0044] Wherein, M represents the number of categories, y ic represents the symbol function, which takes 1 when the real category of sample i is equal to c, and 0 otherwise, p ic represents the predicted probability that the observed sample i belongs to category c.

[0045] The weights of the loss function correspond to the scales of the output image, which are 1, 0.8, 0.6, 0.4 and 0.2 respectively.

[0046] The total loss function is obtained by weighting and fusing the loss functions with different weights, and the optimal segmentation model is obtained by reducing the total loss function.

[0047] As a further improvement of an embodiment of the application, the method further comprises describing the size of the lesion based on the segmentation result, including,

[0048] Based on the segmentation result, the lesion edge contour point is obtained, the straight line equation of the main direction of the lesion is calculated, and the two intersection points of the straight line equation of the main direction and the lesion contour are the end points of the long axis of the lesion contour.

[0049] Setting a short axis search range in the line segment between the long axis endpoints, obtaining N candidate point coordinates, respectively calculating the straight line equation perpendicular to the long axis through the candidate point coordinates, and obtaining N groups of candidate short axis endpoints by intersecting the straight line variance with the lesion contour;

[0050] Calculating the included angle between the long and short axes and the horizontal straight line, wherein the smaller the included angle is, the smaller the horizontal axis is, and the larger the included angle is, the larger the vertical axis is; by importing the actual length of a single pixel of the lesion image, the horizontal axis pixel length distance is obtained, and the actual horizontal and vertical axis size of the lesion is calculated.

[0051] As a further improvement of an embodiment of the application, the method further comprises: based on the lesion contour, describing the boundary definition of the lesion, including,

[0052] The lesion contour is expanded to obtain the region of interest of the lesion, gray-scale operation is performed, and the target region of interest image and the corresponding lesion region mask image are obtained;

[0053] The mask image is respectively subjected to erosion and expansion operation to obtain the region inside and outside the edge of the mask image and the edge region of the mask image;

[0054] The edge algorithm is used to obtain the edge image of the region of interest image, and the edge information graph within a certain range of the peripheral region of the lesion contour is obtained in combination with the edge region of the mask image;

[0055] The lesion edge contour points are output on a full zero graph, the non-zero number is counted as the total number of lesion edges, and the non-zero value of the edge information graph is calculated to obtain the edge ratio;

[0056] In combination with the region inside the edge of the mask image and the region of interest image, the image gray mean value within a certain range of the lesion edge contour is calculated; in combination with the region outside the edge of the mask image and the region of interest image, the image gray mean value within a certain range of the lesion edge contour is calculated; the difference between the two mean values is calculated and the absolute value of the result is taken to calculate the edge difference;

[0057] In combination with the edge ratio and the edge difference, it is judged whether the boundary of the lesion is clear.

[0058] As a further improvement of an embodiment of the application, the method further comprises: based on the lesion contour, describing the shape of the lesion, including,

[0059] The convex hull feature of the lesion contour is calculated to obtain the convex hull of the contour and the concave point in each concave furthest from the corresponding convex hull;

[0060] Screening the concave points, calculating the average distance and the maximum distance of the concave points from the convex hull, and judging whether the lesion shape is regular.

[0061] As a further improvement of an embodiment of the present application, the method further comprises: based on the segmentation result, describing the echo type and echo texture of the lesion, comprising,

[0062] The echo type comprises strong echo, high echo, low echo and no echo;

[0063] Calculating the ratio of the average gray value in the lesion contour area and the average gray value in the lesion excluding the lesion area in the circumscribed rectangle area, and judging the echo type based on the ratio;

[0064] Judging the echo texture based on the gray level features inside the lesion;

[0065] Calculating the average gray value and variance of the lesion image, determining the gray threshold of the lesion image, and binarizing the lesion image according to the threshold to obtain the relatively high-brightness area inside the lesion, and obtaining the proportion of the high-brightness area in the lesion image by counting the number of pixels in the high-brightness area and the number of pixels inside the lesion;

[0066] The entropy of the lesion image is expressed as:

[0067]

[0068] Wherein, i is the pixel gray value, 0≤i≤225, p i is the probability of pixel i appearing in the entire image;

[0069] Combining the proportion of the high-brightness area in the lesion image, the variance and entropy of the lesion image to judge whether the echo texture is uniform.

[0070] Compared with the prior art, the liver lesion image description method based on the semantic segmentation network provided by the present application can segment the liver lesion by constructing a lesion segmentation model, and describe based on the segmentation result, improve the accuracy and reliability of liver disease diagnosis, reduce misjudgment and subjective bias; by combining the advantages of deep learning and traditional image processing, the characteristics of both are fully utilized, the performance and stability of the algorithm are improved; the automatic processing and analysis of the ultrasound image are realized, the burden of the doctor is reduced, and the work efficiency is improved; the results of the lesion description provide accurate disease positioning and type judgment, provide better treatment scheme for the doctor, and improve the treatment effect and treatment experience of the patient. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is the overall flowchart of the liver lesion image description method based on the semantic segmentation network of the present application.

[0072] Figure 2 is a schematic diagram of a Ghost module replacing a convolution unit in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0073] Figure 3 is a schematic diagram of a Ghost bottleneck unit in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0074] Figure 4 is a schematic diagram of a lesion segmentation model in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0075] Figure 5 is a schematic diagram of an atrous spatial convolution pooling layer ASPP in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0076] Figure 6 is a schematic diagram of a lesion ultrasound image and a description result in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0077] Figure 7 is a schematic diagram of a size description in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0078] Figure 8 is a lesion target region of interest image and a mask image in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0079] Figure 9 is an eroded and dilated image of a mask image in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0080] Figure 10 is an image of a region inside an edge, a region outside an edge, and an edge of a mask image in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0081] Figure 11 is a schematic diagram of a total number of lesion edges and a schematic diagram of a lesion edge in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0082] Figure 12 is a schematic diagram of a lesion shape description in the liver lesion image description method based on a semantic segmentation network according to the present application.

[0083] Figure 13 is a schematic diagram of an echo texture description in the liver lesion image description method based on a semantic segmentation network according to the present application. DETAILED DESCRIPTION

[0084] The application will be described in detail below with reference to the specific embodiments shown in the drawings. However, these embodiments do not limit the application, and the changes in structure, method, or function made by those of ordinary skill in the art based on these embodiments are included in the protection scope of the application.

[0085] The embodiments of the application will be described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the application and cannot be understood as a limitation of the application.

[0086] In the first embodiment of the application, the application provides a liver lesion image description method based on a semantic segmentation network, as shown in Figure 1 The method comprises the following steps:

[0087] S1: A Unet semantic segmentation network is built based on a lightweight network GhostNet, and a liver lesion segmentation model is constructed;

[0088] S2: A historical liver lesion ultrasound image dataset is obtained, input into the liver lesion segmentation model for training, and an optimal segmentation model is obtained;

[0089] S3: Based on the segmentation result, the features of the lesion image are described.

[0090] In one specific embodiment of the application, the Ghost module based on the lightweight network GhostNet is used to build the convolution units in the encoder and decoder of the Unet semantic segmentation network, specifically,

[0091] Unet is a deep learning network architecture for semantic segmentation tasks, which consists of an encoder and a decoder. The encoder part is similar to a common convolutional neural network, which gradually reduces the spatial size of the image through a series of convolution and pooling operations, and extracts high-level features of the image. The decoder part gradually recovers the spatial size of the image through upsampling and convolution operations, and fuses the features extracted by the encoder with the features of the decoder, and finally generates a pixel-level semantic segmentation result. GhostNet realizes efficient image classification and target detection in a resource-limited environment through the combination of Ghost module and other lightweight technologies.

[0092] Specifically, the Ghost module can be used to extract more image features, as shown in Figure 2 The implementation method thereof comprises,

[0093] The input information X is defined as R c×h×w , wherein R is the real number field, c is the input channel, h and w are the height and width respectively;

[0094] The input information is obtained through a convolution kernel f' with m filters and a kernel size of k*k, and m output feature maps Y' are obtained, and the output feature map Y' is represented as:

[0095] Y'=X*f'

[0096] The m output feature maps Y' are obtained through linear transformation to obtain n output feature maps Y, wherein mxs=n, n>m, and Y is the output feature of the Ghost module;

[0097] The formula of linear transformation is represented as:

[0098]

[0099] Wherein, y i ′ is the i-th feature map in Y', and Φ i,j (y i ′) represents that the i-th feature map is subjected to the j-th linear transformation to generate the j-th feature map y ij .

[0100] In one specific embodiment of the present application, a Ghost bottleneck unit is constructed based on the Ghost module, as shown in the formula (1), and specifically, Figure 3

[0101] The Ghost bottleneck unit is composed of a first Ghost module, an attention module and a second Ghost module in series, and there is a skip link from the input end to the output end;

[0102] The first Ghost module is used to expand the channel number, the attention module is used to strengthen the feature extraction capability of the Ghost module, and the second Ghost module is used to reduce the channel number;

[0103] Batch normalization and Hard-swish activation function are applied after the first Ghost module, and batch normalization is applied after the second Ghost module;

[0104] The activation function is represented as:

[0105]

[0106] ​It should be noted that batch normalization (Batch Normalization) is used to normalize the input of each layer of the network to zero mean and unit variance, which helps to accelerate the convergence of the model and improve the stability and robustness of the model. By using the BN layer and the Hard-swish activation function in combination, the BN layer can alleviate the problem of gradient disappearance, accelerate the convergence of training, and make the input distribution of the Hard-swish more stable. The Hard-swish activation function has a smoother curve while retaining the nonlinearity of ReLU, which helps to improve the performance and generalization ability of the model.

[0107] The attention mechanism module is added in the Ghost bottleneck unit to strengthen the feature extraction capability of the encoder. The attention mechanism module can use typical SE (Squeeze-and-Excitation Networks), SK (Selective Kernel Networks) and other structures. In actual use, the addition of the attention mechanism can make the lesion contour of the segmentation more complete and conformable.

[0108] In one specific embodiment of the present application, the lesion image is input into the liver lesion segmentation model at different scales, and the feature image is extracted by the corresponding scale encoder, and then the corresponding scale encoder feature image decoder is sequentially passed through to obtain the final output result. Specifically,

[0109] As shown in Figure 4 , the scales of the encoder and the decoder include 1, 1 / 2, 1 / 4, 1 / 8 and 1 / 16; the encoder is composed of 2 Ghost bottleneck units to form a stage, and there are 5 stages in total, and the feature map is down-sampled by 2 times every time it passes through a stage; the decoder is composed of 2 Ghost bottleneck units to form a stage, and there are 5 stages in total, and the feature map is up-sampled by 2 times every time it passes through a stage.

[0110] In one specific embodiment of the present application, an atrous spatial convolution pooling layer ASPP is added to the input layer of the liver lesion segmentation model, wherein the feature image with a scale of 1 / 16 after encoding needs to pass through the atrous spatial convolution pooling layer ASPP before passing through the decoder, and the feature images with other scales are subjected to channel dimension splicing operation;

[0111] Specifically, as shown in Figure 5As shown, the Spatial Dilated Convolutional Pooling (ASPP) layer consists of four parallel dilated convolutions and one global average pooling layer. The four parallel dilated convolutions include a 1x1 convolution, a 3x3 convolution with a dilation factor of 3, a 3x3 convolution with a dilation factor of 6, and a 3x3 convolution with a dilation factor of 9. The global average pooling layer upsamples the data through a 1x1 convolution operation. The feature images obtained from the four parallel dilated convolutions and the upsampled data are concatenated and then processed through a 1x1 convolution operation to obtain the final output. The final output includes receptive fields at different levels.

[0112] In one specific embodiment of the present invention, the liver lesion segmentation model is trained, specifically by:

[0113] Specifically, the process begins with collecting historical data on liver lesions. This historical data dataset should include as many liver lesions as possible (cysts, hemangiomas, liver cancer, hyperechoic lesions, etc.), and undergo preprocessing such as cleaning, augmentation, and enhancement. Data collection should be conducted from multiple representative medical institutions across different regions and levels to ensure data diversity and a relatively even distribution of lesion images across various disease types. Further, the collected data is cleaned, removing outdated or low-quality data. Additionally, the data is desensitized, removing patient and hospital-related information from the images. Next, ultrasound images containing lesions are cropped online within a certain range based on the region of interest (ROI) of the lesion, using random pixel increments or lesion size ratios, to obtain training data. This data is then augmented online to obtain training data with more varied scenarios. Augmentation operations include, but are not limited to, horizontal and vertical flipping with a certain probability, random rotation, cropping, translation, pixel value perturbation, filtering, and noise addition. The ROI of the lesion can be obtained through a pre-built target detection model or manually defined.

[0114] The liver lesion images that meet the training requirements are input into the liver lesion segmentation model for training. The output of the segmentation model adopts a deep supervision form with multiple segmentation heads. Different loss functions are set for different segmentation heads according to their different positions. Finally, these loss functions are fused together as the final optimization target.

[0115] like Figure 4 As shown, in the network structure of the segmentation model, Down-Sample 2*(4*, 8*, 16*) represents downsampling by 2x, 4x, 8x, and 16x respectively, Up-Sample 2* represents upsampling by 2x, Ghostbottleneck*2 represents using 2 Ghostbottleneck units, Concatenate represents concatenating feature maps along the channel dimension, and ASPP represents dilated spatial convolutional pooling layer.

[0116] Specifically, the Dice-Loss loss function and the cross-entropy loss function are used for the output result with the scale of 1, and the cross-entropy loss function is used for the output result with other scales;

[0117] The Dice-Loss loss function is represented as:

[0118]

[0119] wherein X represents the pixel label of the real segmentation image, Y represents the pixel category of the model predicted segmentation image, |X| and |Y| represent the number of elements in X and Y respectively, and |X∩Y| represents the number of intersection elements between X and Y;

[0120] The cross-entropy loss function is represented as:

[0121]

[0122] wherein M represents the number of categories, y ic represents the symbol function, and takes 1 when the real category of the sample i is equal to c, and otherwise takes 0, p ic represents the predicted probability that the observed sample i belongs to the category c, Cross-Entropy-Loss represents the cross-entropy loss function, the cross-entropy is used to evaluate the difference between the probability distribution obtained by the current training and the real distribution, and represents the distance between the actual output (probability) and the expected output (probability), and the smaller the cross-entropy value is, the closer the two probability distributions are.

[0123] The weights of the loss functions correspond to the scales of the output images, and are 1, 0.8, 0.6, 0.4 and 0.2 respectively;

[0124] The total Loss loss function is obtained by weighting and fusing the loss functions with different weights, and the optimal segmentation model is obtained by reducing the total Loss loss function.

[0125] In one specific embodiment of the present application, as shown in Figure 6 , it is an ultrasound image of a liver lesion, based on the segmentation result, the size of the lesion is described, as shown in Figure 7 , specifically,

[0126] Based on the segmentation result, the straight line equation of the main direction of the lesion is obtained, and the two intersection points of the straight line equation of the main direction and the lesion contour are the end points of the long axis of the lesion contour;

[0127] Specifically, the straight line equation of the main direction of the lesion is calculated by using the PCA principal component analysis method, wherein PCA is an unsupervised algorithm, which can well find the direction of the sample.

[0128] A short axis search range is set in the line segment between the long axis end points, N candidate point coordinates are obtained, the equation of a straight line perpendicular to the long axis is calculated through the candidate point coordinates, and the straight line intersects with the lesion contour to obtain N groups of candidate short axis end points; the pixel length of the candidate short axis is calculated, and the one with the minimum distance value is the short axis of the lesion;

[0129] Specifically, the short axis search range is set from 1 / 3 of the short axis line segment to 2 / 3.

[0130] Preferably, in the actual selection range of the short axis point, in order to obtain a short axis that is more in line with the actual situation, the equation of a straight line is calculated, which is not only perpendicular to the long axis, but also has an angle of 81 degrees, 84 degrees, 87 degrees, 93 degrees, 96 degrees or 99 degrees with the long axis in a clockwise direction. When there are multiple groups of minimum short axes with equal distances, the short axis closest to the center of the long axis and most perpendicular to the long axis is selected.

[0131] Further, the angle between the long and short axes and a horizontal straight line is calculated, wherein the smaller the angle, the smaller the horizontal axis, and the larger the angle, the larger the vertical axis; the actual horizontal and vertical axis sizes of the lesion are obtained by introducing the actual length of a single pixel of the lesion image to obtain the horizontal axis pixel length distance.

[0132] In one specific embodiment of the present application, the boundary sharpness of the lesion is described based on the lesion contour, specifically,

[0133] As shown in Figure 8 , the lesion contour is dilated to obtain a region of interest of the lesion, gray-scale processing is performed, and a target region of interest image ROI_Image and a corresponding lesion region mask image Mask_Image are obtained;

[0134] As shown in Figure 9 , the mask image is subjected to erosion and dilation operations respectively to obtain an eroded image Mask_Erode and a dilated image Mask_Di late. As shown in Figure 10 , the region inside the edge of the mask image Mask_In is obtained by subtracting the eroded image Mask_Erode from the mask image Mask_Image, the region outside the edge of the mask image Mask_Out is obtained by subtracting the mask image Mask_Image from the dilated image Mask_Di late, and the edge region Mask_Edge of the mask image is obtained by adding the region inside the edge of the mask image Mask_In and the region outside the edge of the mask image Mask_Out.

[0135] An edge algorithm is used to obtain an edge image Mask_Canny for the image ROI_Image, and an edge information image Mask_Canny_Edge within a certain range of the edge region of the mask image Mask_Edge is obtained;

[0136] Preferably, the edge algorithm used in the present application is a Canny edge algorithm, wherein the low threshold value required by the Canny algorithm is set to 30, and the high threshold value is set to 180.

[0137] As shown in Figure 11 the edge profile points of the lesion are output on a full zero image, the non-zero number is counted as the total number of edge profiles EdgeNumber, and the non-zero value DefinitionEdgeNumber of the edge information image Mask_Canny_Edge is calculated, and the edge ratio DefinitionEdgeRatio is calculated;

[0138]

[0139] The image gray mean InnerAvg within a certain range of the edge profile of the lesion is calculated by combining the image ROI_Image and the region Mask_In within the edge of the mask image, the image gray mean OuterAvg within a certain range of the edge profile of the lesion is calculated by combining the image ROI_Image and the region Mask_Out outside the edge of the mask image, the difference between the two means is calculated and the absolute value of the result is taken to calculate the edge difference DefinitionEdgeDiff;

[0140] |InnerAvg-OuterAvg|=Definit ionEdgeDiff

[0141] The edge ratio and the edge difference are combined to determine whether the boundary of the lesion is clear. Specifically, when DefinitionEdgeDiff≥10, the edge is clear; when DefinitionEdgeDiff<10 and DefinitionEdgeRatio>0.19, the edge is clear; and in other cases, the edge is not clear.

[0142] In one specific embodiment of the present application, the shape of the lesion is described based on the lesion profile, specifically,

[0143] The convex hull feature of the lesion profile is calculated to obtain the convex hull of the profile and the concave point in each concave region farthest from the corresponding convex hull;

[0144] Preferably, as shown in Figure 12As shown, by calling the convex hull detection algorithm in the open-source image processing library, the convex hull of the contour is obtained, as shown by the yellow contour in the figure, and the contour coordinates point farthest from the corresponding convex hull in each depression is obtained, as shown by the red dot in the figure.

[0145] The indented points are selected, the average and maximum distances between the indented points and the convex hulls are calculated, and it is determined whether the shape of the lesion is regular.

[0146] Furthermore, a threshold of 5 pixels is set to filter these concave points, and then the number of concave points that meet the conditions (ConcaveNum), the average distance (ConcaveAvgDepth) and the maximum distance (ConcaveMaxDepth) between the concave points and the convex hull are calculated. In addition, to eliminate the influence of lesions of different sizes on the calculation results, ConcaveAvgDepth and ConcaveMaxDepth are divided by the lesion minor axis obtained above, so as to enable the algorithm to have adaptive capabilities. Logical judgment is performed on the above calculated values ​​to obtain the lesion shape description. Specifically, when ConcaveAvgDepth > 0.1 or ConcaveMaxDepth > 0.22, the shape description is irregular; otherwise, it is regular.

[0147] In one specific embodiment of the present invention, based on the segmentation results, the echo type and echo texture of the lesion are described, specifically,

[0148] Echo types include strong echo, high echo, low echo, and anechoic;

[0149] Calculate the ratio R between the mean gray value M1 within the lesion contour area and the mean gray value M2 within the lesion's outer rectangular area after removing the lesion area, and determine the echo type based on the ratio;

[0150]

[0151] Specifically, on the one hand, when R>1.1, it is a high echo or strong echo; further, when R<1.5, it is a high echo; when R>2 and M1<80, it is a high echo, and vice versa. On the other hand, when R≤1.1, it is a low echo or no echo; further, when R>0.7, it is a low echo, and vice versa.

[0152] like Figure 13 As shown, the echo texture is determined based on the grayscale characteristics inside the lesion;

[0153] The gray mean value Mean and the variance Std of the lesion image are calculated, the gray threshold Threshold of the lesion image is determined, and the relative highlight area in the lesion is obtained by binarizing the lesion image according to the threshold, and the proportion of the highlight area in the lesion image is obtained by counting the number of pixels of the highlight area LightSpotsNum and the number of pixels in the lesion LesionPixelNum, and the proportion of the highlight area in the lesion image is LightSpotsRatio.

[0154] It should be noted that the gray mean value reflects the average gray level of the lesion image, and the variance reflects the degree of change of the gray value in the image.

[0155] The calculation formula of the threshold is:

[0156] Threshold=Mean=K×Std

[0157] Wherein, K is an empirical weight value, which is 1.8 in the present application.

[0158] The entropy of the lesion image is represented as:

[0159]

[0160] Wherein, i is the pixel gray value, 0≤i≤225, p i is the probability of pixel i appearing in the whole image, which is represented as:

[0161]

[0162] It should be noted that the entropy in the lesion area is calculated, and the entropy of the digital image refers to the information amount contained by the aggregation characteristics of the gray distribution in the image, which can reflect the complexity of the texture in the image. The greater the entropy, the more complex the texture, and the smaller the entropy, the smoother the texture.

[0163] Further, the proportion of the highlight area in the lesion image LightSpotsRatio, the variance Std and the entropy H of the lesion image are combined to judge whether the echo texture is uniform. Specifically, when H>5 and Std>25, the echo texture is not uniform; otherwise, further judgment is made, when LightSpotsRatio>0.035, the echo texture is not uniform; otherwise, the echo texture is uniform.

[0164] In summary, the liver lesion image description method based on the semantic segmentation network provided by the application can improve the accuracy and reliability of liver disease diagnosis, reduce misjudgment and subjective bias by constructing a lesion segmentation model to segment liver lesions and describing based on the segmentation result; by combining the advantages of deep learning and traditional image processing, the characteristics of both are fully utilized to improve the performance and stability of the algorithm; automatic processing and analysis of ultrasound images are realized, which reduces the burden of doctors and improves work efficiency; the lesion description result provides accurate disease positioning and type judgment, which provides doctors with a better treatment plan and improves the treatment effect and treatment experience of patients.

[0165] It should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

[0166] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described modules can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0167] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, that is, they can be located in one place or distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0168] In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can exist physically alone, or 2 or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of hardware plus software functional module.

[0169] The integrated module realized in the form of the software function module can be stored in a computer readable storage medium. The software function module is stored in a storage medium, and includes a plurality of instructions for enabling a computer system (which can be a personal computer, a server, or a network system, etc.) or a processor to execute part of steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A liver lesion image description method based on a semantic segmentation network, characterized in that, The application relates to a liver lesion segmentation method based on a lightweight network GhostNet. The application comprises the following steps: A liver lesion segmentation model is constructed based on a Unet semantic segmentation network of the GhostNet; An optimal segmentation model is obtained by training the liver lesion segmentation model based on a historical liver lesion ultrasound image dataset; Features of a lesion image are described based on a segmentation result; 2.The liver lesion image description method based on the semantic segmentation network according to claim 1, characterized in that, The GhostNet comprises a Ghost module, a Ghost bottleneck unit is constructed based on the Ghost module, and the liver lesion segmentation model is constructed based on the Ghost bottleneck unit; the Ghost bottleneck unit is composed of a first Ghost module, an attention module and a second Ghost module in series, and a skip link exists from an input end to an output end; the first Ghost module is used for expanding a channel number, the attention module is used for strengthening feature extraction capability of the Ghost module, and the second Ghost module is used for reducing the channel number; batch normalization and a Hard-swish activation function are applied after the first Ghost module, and batch normalization is applied after the second Ghost module. The Unet semantic segmentation network is composed of a symmetrical down-sampling path encoder and an up-sampling path decoder; Convolution units in the encoder and the decoder of the Unet semantic segmentation network are constructed based on a Ghost module of the GhostNet; The Ghost module is used for extracting more image features, Defining input information wherein is the real domain, is the input channel, and are the height and width, respectively; The input information is transmitted through a device with... There are 1 filter, and the kernel size is 1. convolution kernel ,get Output feature map The output feature map Represented as: The output feature maps are obtained by linear transformation output feature maps wherein , are the output features of the Ghost module The application comprises the following steps: wherein, is the feature map, denotes performing the linear transformation on the feature map to generate the feature map . 3.The liver lesion image description method based on the semantic segmentation network of claim 1, wherein, The formula of the linear transformation is represented as: The application further comprises the following steps: 。 4. The liver lesion image description method based on a semantic segmentation network according to any one of claims 1, characterized in that, The activation function is represented as: Lesion images are input into the liver lesion segmentation model in different scales, feature images are extracted by corresponding encoders, and finally output results are obtained by sequentially passing through feature image decoders of the corresponding scale encoders; The scales of the encoder and the decoder include 1, 1 / 2, 1 / 4, 1 / 8 and 1 / 16; The encoder is composed of 2 Ghost bottleneck units to form one stage, and there are 5 stages in total, and a feature map is down-sampled by 2 times per stage; 5. The liver lesion image description method based on a semantic segmentation network according to claim 4, characterized in that, The decoder is composed of 2 Ghost bottleneck units to form one stage, and there are 5 stages in total, and a feature map is up-sampled by 2 times per stage. A spatial convolutional pooling layer ASPP is added to an input layer of the liver lesion segmentation model; Wherein, a feature image with a scale of 1 / 16 needs to pass through the spatial convolutional pooling layer ASPP before passing through the decoder, and feature images with other scales are subjected to a channel dimension splicing operation; The spatial convolutional pooling layer ASPP comprises 4 parallel spatial convolutional and 1 global average pooling; The 4 parallel spatial convolutional comprise 1*1 convolution, 3*3 convolution with an expansion coefficient of 3, 3*3 convolution with an expansion coefficient of 6 and 3*3 convolution with an expansion coefficient of 9; The global average pooling is up-sampled by 1*1 convolution operation. The four parallel empty convolution feature images and the up-sampling are spliced together, and a final output is obtained through a 1*1 convolution operation. The final output includes different levels of receptive fields. 6.The liver lesion image description method based on semantic segmentation network according to claim 5, characterized in that, The liver lesion segmentation model is trained, including, The Dice-Loss loss function and the cross-entropy loss function are used for the output scale of 1, and the cross-entropy loss function is used for other scale output results; The Dice-Loss loss function is represented as: wherein, represents the pixel labels of the real segmentation image, represents the pixel classes of the model predicted segmentation image, and respectively represent , the number of elements in represents and the number of intersection elements between and The cross-entropy loss function is represented as: wherein, denotes the number of classes, denotes the sign function, which takes the value 1 when the sample has a true class equal to and 0 otherwise, denotes the predicted probability that the observed sample belongs to class ; The weights of the loss function correspond to the scales of the output image, which are 1, 0.8, 0.6, 0.4 and 0.2 respectively; The total Loss loss function is obtained by weighting and fusing loss functions with different weights, and the optimal segmentation model is obtained by reducing the total Loss loss function. 7.The liver lesion image description method based on semantic segmentation network according to claim 1, characterized in that, Based on the segmentation result, the size of the lesion is described, including, Based on the segmentation result, the edge contour points of the lesion are obtained, the straight line equation of the main direction of the lesion is calculated, and the two intersection points of the straight line equation and the lesion contour are the end points of the long axis of the lesion contour. A short axis search range is set in the line segment between the long axis end points to obtain N candidate point coordinates, the straight line equation perpendicular to the long axis is calculated through the candidate point coordinates, and N groups of candidate short axis end points are obtained by intersecting the straight line equation with the lesion contour. The pixel length of the candidate short axis is calculated, and the one with the smallest distance value is the short axis of the lesion. The included angle between the long and short axes and the horizontal straight line is calculated, wherein the smaller the included angle, the smaller the horizontal axis, and the larger the included angle, the larger the vertical axis. The actual length of a single pixel of the lesion image is imported to obtain the horizontal axis pixel length distance, and the actual horizontal and vertical axis size of the lesion is calculated.

8. The liver lesion image description method based on a semantic segmentation network according to claim 7, characterized in that, Based on the lesion contour, the boundary definition of the lesion is described, including, The lesion contour is expanded to obtain the region of interest of the lesion, the grayscale operation is performed, and the target region of interest image and the corresponding lesion region mask image are obtained. The mask image is respectively subjected to erosion and expansion operations to obtain the region inside and outside the edge of the mask image and the edge region of the mask image; The edge algorithm is used to calculate the edge image of the region of interest image, and the edge information graph within a certain range of the lesion contour peripheral region is obtained in combination with the edge region of the mask image; The lesion edge contour points are output on a full zero graph, and the non-zero number is counted as the total number of lesion edges, and the non-zero value of the edge information graph is calculated. The image grayscale mean value within one circle range of the lesion edge contour is calculated in combination with the region inside the edge of the mask image and the region of interest image; the image grayscale mean value within one circle range outside the lesion edge contour is calculated in combination with the region outside the edge of the mask image and the region of interest image; the difference between the two mean values is calculated, and the absolute value of the result is taken to calculate the edge difference. The edge ratio and the edge difference are combined to determine whether the boundary of the lesion is clear. 9.The liver lesion image description method based on semantic segmentation network according to claim 7, characterized in that, Based on the lesion contour, the shape of the lesion is described, including, The convex hull feature of the lesion contour is calculated to obtain the convex hull of the contour and the concave point in each concave that is farthest from the corresponding convex hull. Screening the concave points, calculating the average distance and the maximum distance of the concave points from the convex hull, and judging whether the lesion shape is regular. 10.The liver lesion image description method based on semantic segmentation network according to claim 1, characterized in that, Based on the segmentation result, the echo type and echo texture of the lesion are described, including, The echo type includes strong echo, high echo, low echo and no echo; The ratio of the average gray value in the lesion contour area to the average gray value in the lesion excluding the lesion area is calculated, and the echo type is judged based on the ratio; The echo texture is judged based on the gray level characteristics inside the lesion; The gray level mean and variance of the lesion image are calculated, the gray level threshold of the lesion image is determined, and the lesion image is binarized according to the threshold to obtain the relatively high-light area inside the lesion. The proportion of the high-light area in the lesion image is obtained by counting the number of pixels in the high-light area and the number of pixels inside the lesion. The entropy of the lesion image is expressed as: wherein, is the pixel gray value, , is the pixel probability of occurrence in the whole image; The echo texture is judged whether it is uniform or not in combination with the proportion of the high-light area in the lesion image, the variance and the entropy of the lesion image.

Images