A method for establishing an artificial intelligence assisted ultrasound diagnosis of a liver and spleen trauma model

Abstract

The present application relates to a kind of artificial intelligence auxiliary ultrasonic diagnosis method for establishing spleen and liver trauma model, belong to artificial intelligence auxiliary diagnosis field.The present application uses the method of data enhancement and generation, the ultrasound image of animal spleen and liver trauma is enhanced to more close to the strong data set of human spleen and liver trauma clinical data by data;Then a large number of enhanced ultrasound image of spleen and liver trauma is created source model, and fine tuning is carried out under a small amount of human spleen and liver trauma data using source model;Finally, the model of spleen and liver trauma ultrasound image automatic segmentation and the model of grade classification of spleen and liver trauma ultrasound image are established, the image generated in the process of spleen and liver trauma ultrasonic diagnosis is analyzed, the contour of the anatomical structure of spleen and liver appearing in the segmented image and the position of spleen and liver trauma are identified, and grade classification result is obtained.The present application shows good performance on clinical spleen and liver trauma ultrasound image data set.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence auxiliary diagnosis, and particularly relates to a method for establishing an artificial intelligence auxiliary ultrasound diagnosis spleen and liver trauma model. BACKGROUND

[0002] Abdominal injury is common in traffic accidents, air crashes, industrial labor accidents, and knife wounds, gunshots, etc. in fights, and is often part of multiple injuries due to the large number of abdominal organs, which can easily cause massive hemorrhage and severe infection, shock and respiratory failure, and has a high mortality rate. Closed abdominal injury is often accompanied by severe problems of organs due to the early symptoms of some patients not being obvious, and requires more timely and accurate diagnosis. Liver and spleen are both abdominal solid organs, and are also prone to traumatic liver and spleen rupture when subjected to external force. Among them, the spleen is the most vulnerable organ due to its fragile texture and rich blood supply, and spleen trauma accounts for 13-25% of abdominal multiple injuries, and can endanger the life of a patient if not diagnosed and treated in time. The incidence of liver injury is second only to that of the spleen, and the condition of severe liver injury is complex, with many complications and a high mortality rate. Therefore, exploring the diagnosis and pretreatment of such injuries is still an important issue in current abdominal injuries.

[0003] In the past, surgical procedures such as diagnostic peritoneal lavage and laparotomy were usually used to diagnose abdominal injuries. But this often causes unnecessary trauma and increases the risk of complications such as infection for some hemodynamically stable patients. Currently, preliminary screening by imaging has become the main way to quickly assess the injury. Imaging methods mainly include CT examination and ultrasonography. The advantage of CT examination is high coincidence rate, and it is not affected by gas interference, patient breathing, etc. during the examination. However, CT cannot be moved at will due to its instrument conditions, so it cannot be used for pre-hospital and emergency bedside patients. In addition, CT has radioactivity, which increases the risk of exposure to radiation for children and pregnant women, limiting the examination. The thickness of the CT scan layer may also miss some small lesions. Therefore, CT scanning has limitations. Ultrasonography can be used for rapid injury assessment in pre-hospital and emergency departments due to its convenient operation, easy portability, and low operation difficulty. Ultrasonography can determine whether there is organ damage by observing the continuity of the splenic capsule, the presence or absence of subcapsular hematoma, and the uniformity of the parenchymal echo. At the same time, ultrasonography can also infer whether there is organ damage through some indirect signs, such as retroperitoneal hematoma and ascites. Color Doppler ultrasonography can detect abnormal blood supply in lesions. However, the diagnostic sensitivity of conventional ultrasonography for parenchymal organ damage is only about 41%. Contrast-enhanced ultrasound (CEUS) can detect parenchymal organ damage and active bleeding in abdominal multiple injuries, improve the accuracy of ultrasonography in detecting splenic injury, and provide a more reliable assessment method for parenchymal organ damage, making up for the shortcomings of conventional ultrasonography. Studies have shown that CEUS is as accurate as CT in detecting and staging traumatic splenic injury. However, CEUS is an invasive examination, and the contrast agent is expensive, so the examination has limitations.

[0004] Artificial intelligence, as a strategic technology leading the future, is increasingly applied in various fields of society. In 2016, the world's first artificial intelligence ultrasound series product was released, and artificial intelligence began to be applied in the diagnosis of thyroid nodules and could improve the accuracy of thyroid nodule diagnosis. Deep learning, as one of the popular analysis methods of machine learning at present, can learn the representation of massive data through the construction of multi-layer artificial neural networks. The rapid growth of graphics processing capability makes it possible to develop the most advanced algorithms, which have more stable and powerful image analysis capabilities.

[0005] The imaging principle of medical images and the characteristics of human tissues themselves make medical images different from ordinary natural images, with more blurred and uneven characteristics. Noise, field offset effect, local volume effect and tissue movement affect the formation of the image. The structure and shape of anatomical tissues are extremely complex, and pathological tissues often change the original shape of the target object, with large individual differences, which brings difficulties to medical image segmentation. At the same time, since the segmentation result of medical images ultimately serves medical diagnosis, and is related to human health and life, ignoring its subtle differences can seriously affect the diagnosis result, so there is a high requirement for its segmentation accuracy. Accurate segmentation can reduce errors for subsequent diagnosis steps, thereby reducing the risk of misjudgment, and medical image segmentation algorithms based on artificial intelligence can fully learn limited medical image data, and can play a high-performance advantage in the medical image field to provide good segmentation results for reference. Therefore, it has been widely used in the diagnosis and identification of various organ ultrasound images and has been proven to have high accuracy.

[0006] Currently, there are few reports on artificial intelligence assisted ultrasound diagnosis of splenic and hepatic trauma, and due to the particularity of splenic and hepatic trauma, the number of splenic and hepatic trauma cases is limited, and there is a lack of large-scale splenic and hepatic trauma clinical ultrasound image dataset. However, deep learning models require a large amount of medical image data, in order to solve this problem, we use data enhancement and generation methods, first, the ultrasound images of splenic and hepatic trauma animals are enhanced through data enhancement to form a strong dataset closer to clinical data, then a large number of data enhanced ultrasound images of splenic and hepatic trauma animals are applied to create a source model, and then fine-tuned under a small amount of human clinical data, finally, an artificial intelligence assisted ultrasound diagnosis model of splenic and hepatic trauma is established. Therefore, this study combines animal experiments and clinical ultrasound image data, and applies deep learning methods to establish an artificial intelligence assisted ultrasound diagnosis model of splenic and hepatic trauma.

[0007] Madalin Mamuleanu et al. proposed a technical solution for liver ultrasound image tumor segmentation based on U-Net in the article "LIVER LESION SEGMENTATION IN CONTRAST-ENHANCED ULTRASOUND USING DEEP LEARNING ALGORITHMS". The purpose of the deep learning algorithm used is to segment ultrasound contrast video examinations to identify lesions. First, in all frames of 50 uncompressed video recordings of ultrasound contrast examinations, b-mode regions are defined and extracted, and at the same time, senior gastroenterologists manually segment lesions in each frame of image to form a dataset required by the model, including b-mode images and their corresponding masks. Second, a U-Net version with a tensor size of 256x256 is used for training and prediction.

[0008] Step 1: data collection and dataset processing: in all frames of the uncompressed video recording of 50 ultrasound contrast examinations, the b-mode region is defined and extracted, and the lesions in each frame of image are manually segmented by a senior gastroenterologist to form the dataset required for modeling, including b-mode images and their corresponding masks. The dataset is randomly divided into 70% for training, 20% for testing, and 10% for verification.

[0009] Step 2: model construction and training: a U-Net version with a tensor size of 256x256 is used to train the U-Net model with an Adam optimizer with a batch size of 8 for 50 epochs, which finds a separate learning rate for each parameter.

[0010] The U-net network uses a classic encoder-decoder structure, and the network consists of two parts: the encoding stage uses a contracting path to gradually reduce the spatial dimensions of the feature map to obtain high-level context information, and the decoding stage uses a symmetric expanding path corresponding to upsampling to restore the resolution. In addition, the skip connection that passes the encoder output to the corresponding level decoder reduces the spatial information loss brought by the downsampling process. The model finally established in this technical solution has a total of 412,865 trainable parameters.

[0011] Step 3: test performance: the intersection over union (IoU), recall rate and precision are used to measure the performance of the model, and the performance test results are IoU=0.7, recall rate=0.76, and precision=0.86.

[0012] In this prior art solution, there are several problems that are not suitable for the required scene of the present application:

[0013] 1. The application scenario is tumor segmentation of liver ultrasound images, not trauma site segmentation of liver or even spleen.

[0014] In the required scene of the present application, the spleen and liver organs and the trauma location need to be accurately located, and the trauma classification results are further obtained accordingly.

[0015] 2. The number of ultrasound image samples is small, and the data set generated by the data collection method is small, which is not conducive to model training.

[0016] In the required scene of the present application, the segmentation result accuracy of organs and trauma locations has a very high requirement, and ignoring the subtle differences will seriously affect the diagnosis result, and accurate segmentation can often reduce errors for grade classification and subsequent diagnosis steps, thereby reducing the risk of misjudgment.

[0017] 3、The model adopted is too simple and cannot fully extract the context information in the spleen and liver ultrasound images.

[0018] Due to the limitation of the convolution structure in the U-Net network, it is difficult for the algorithm based on convolution alone to learn the global context information. In the scene required by the present application, in order to better extract the context information, it is necessary to combine global and local context information for feature extraction and fusion. SUMMARY

[0019] (1) Technical problems to be solved

[0020] The technical problem to be solved by the present application is how to provide an artificial intelligence assisted ultrasound diagnosis spleen and liver trauma model establishment method to solve the problems of difficulty in collecting a large amount of image data of spleen and liver trauma, lack of auxiliary analysis model of spleen and liver trauma, and inability of existing segmentation models to be applied to spleen and liver trauma segmentation.

[0021] (2) Technical solutions

[0022] In order to solve the above technical problems, the present application provides an artificial intelligence assisted ultrasound diagnosis spleen and liver trauma model establishment method, which comprises the following steps:

[0023] S21, obtaining an ultrasound image data set of animal spleen and liver trauma as a weak data set, and obtaining a human spleen and liver trauma ultrasound image data set as a strong data set;

[0024] S22, using a convolutional neural network model and a data generation network to enhance the weak data set of animal spleen and liver trauma ultrasound images with the strong data set of human spleen and liver trauma ultrasound images, to obtain a data set generated after data enhancement of the weak data set of animal spleen and liver trauma ultrasound images, to establish a pre-training data set, and using the human spleen and liver trauma ultrasound image data set to establish a fine-tuning data set, wherein the pre-training data set and the fine-tuning data set each include normal non-trauma images and trauma images;

[0025] S23, artificially labeling the pre-training data set and the fine-tuning data set respectively to form corresponding artificial labeling segmentation label data sets and trauma grade classification label data sets; wherein the labels of the artificial labeling segmentation label data set include background, spleen / liver structure, and spleen / liver trauma;

[0026] S24, establishing a spleen and liver trauma ultrasound image automatic segmentation model based on the pre-training data set, the fine-tuning data set, and the artificially labeled segmentation label data set, and using the spleen and liver trauma ultrasound image automatic segmentation model to process clinical spleen and liver trauma ultrasound images to obtain segmentation prediction results, the segmentation prediction results including spleen and liver positions and spleen and liver contours, and injured parts and / or injured contour information;

[0027] S25, based on the pre-training dataset, the fine-tuning dataset and the corresponding artificial labeled trauma grade classification label dataset, a spleen and liver trauma ultrasound image grade classification model is established by training, and the segmentation prediction result output in step S24 is predicted using the spleen and liver trauma ultrasound image grade classification model to obtain the corresponding spleen and liver trauma grade classification label.

[0028] (III) Beneficial effects

[0029] The application proposes a method for establishing an artificial intelligence assisted ultrasound diagnosis spleen and liver trauma model. The application adopts a data enhancement and generation method to train a spleen and liver trauma ultrasound image automatic segmentation model and a spleen and liver trauma ultrasound image grade classification model. First, the ultrasound images of animal spleen and liver trauma are combined with a residual learning convolutional neural network, and data enhancement is used to obtain a strong dataset closer to clinical data, thereby obtaining a larger scale spleen and liver trauma ultrasound image dataset. Then, the animal spleen and liver trauma ultrasound image dataset and the corresponding artificial labeled segmentation label dataset and grade classification label dataset are used for pre-training, and the clinical spleen and liver trauma ultrasound image dataset and the corresponding artificial labeled segmentation label dataset and grade classification label dataset are used for fine-tuning, so as to finally establish an artificial intelligence assisted ultrasound diagnosis spleen and liver trauma model.

[0030] The role of data enhancement and generation is to learn the distribution rules of different training datasets in the same class from the feature distribution of strong training datasets, extract image features, structural features and semantic features, and use the feature distribution rules contained in the strong dataset to enhance the weak training dataset through the data generation network, thereby expanding the training data in the weak dataset and improving the utilization rate of the weak dataset in training.

[0031] The ultrasound diagnosis spleen trauma segmentation model adopted in the application is optimally designed in terms of local context information and multi-scale context information.

[0032] On the basis of the advantage of capturing global context information by using the Transformer, the local context information in the spleen and liver trauma ultrasound image is extracted. Therefore, in the application, the channel block local context information extraction module is used to replace the traditional FFN module in the spleen and liver trauma ultrasound image automatic segmentation model, so as to solve the problem that the global matrix operation in the existing Transformer framework ignores the local context information of details. The local context information is obtained by the channel block local context information extraction module in the application to supplement the semantic information in a small range of regions, improve the segmentation performance on clinical spleen and liver trauma ultrasound images, and excavate high position sensitivity and fine-grained image details.

[0033] On the basis of the advantage of capturing global context information by using the transformer, the multi-scale context information in the spleen and liver trauma ultrasound image is extracted. Therefore, the automatic segmentation model of the spleen and liver trauma ultrasound image in the application uses the skip layer connection based on the U-Net on the basis of the existing transformer framework, that is, the output of the encoder is transmitted to the corresponding level decoder as the input and is spliced with the up-sampling result of the last decoding layer, the spatial information loss caused by the down-sampling process is reduced, the feature map in the channel dimension is spliced, the feature map recovered by the up-sampling contains more low-level semantic information, which is helpful to recover the lesion details and the spatial dimension for accurate positioning, the fusion result across the resolution is obtained, and the representability of the multi-scale is improved. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 It is a data enhancement model structure diagram of the application;

[0035] Figure 2 It is a weak data set enhancement network structure diagram;

[0036] Figure 3 It is a label image schematic diagram before and after one-hot encoding;

[0037] Figure 4 It is an ultrasound diagnosis spleen trauma segmentation model network structure diagram;

[0038] Figure 5 It is a transformer block structure-channel block local context information extraction module;

[0039] Figure 6 It is a spleen and liver trauma grade classification model network structure diagram. DETAILED DESCRIPTION

[0040] In order to make the purpose, content and advantages of the application more clear, the specific embodiments of the application are further described in detail below in combination with the drawings and examples.

[0041] The application aims at providing a kind of artificial intelligence auxiliary ultrasound diagnosis spleen and liver trauma model establishment method, solve the difficulty of a large number of image data acquisition of spleen and liver trauma, lack of auxiliary analysis model of spleen and liver trauma, existing segmentation model cannot be applicable to the problem of spleen and liver trauma segmentation.

[0042] The artificial intelligence auxiliary ultrasound diagnosis spleen and liver trauma model establishment method of the application, it includes the following steps:

[0043] S11, first, the method of data enhancement and generation is used, and the ultrasound image of the animal spleen and liver trauma easily acquired is enhanced to a strong data set closer to the clinical data of human spleen and liver trauma;

[0044] S12, then a large number of enhanced post-spleen and liver trauma ultrasound images are applied to create a source model, since there are differences between the images of the source model and the real human spleen and liver trauma clinical images, therefore, the source model is fine-tuned under a small amount of human spleen and liver trauma data;

[0045] S13, finally, a spleen and liver trauma ultrasound image automatic segmentation model and a spleen and liver trauma ultrasound image grading classification model are established, the model analyzes the images generated in the process of spleen and liver trauma ultrasound diagnosis, identifies the contours of the spleen and liver anatomical structures and the positions of the spleen and liver trauma in the segmented images, and obtains the grading classification results.

[0046] The method provided by the application obtains an artificial intelligence assisted ultrasound diagnosis spleen and liver trauma model based on a Transformer framework, including a spleen and liver trauma ultrasound image automatic segmentation model and a spleen and liver trauma ultrasound image grading classification model, on the basis of the advantage of capturing global context information by the Transformer, the local context information and multi-scale context information in the spleen and liver trauma ultrasound image are fully mined and utilized, and good performance is shown on a clinical spleen and liver trauma ultrasound image data set.

[0047] In order to solve the problems in the existing spleen and liver trauma ultrasound image segmentation method, the application provides a segmentation method for artificial intelligence assisted ultrasound diagnosis of spleen and liver trauma, which comprises the following specific steps:

[0048] S21, obtain the ultrasound image data set of animal spleen and liver trauma as a weak data set, and obtain the human spleen and liver trauma ultrasound image data set as a strong data set.

[0049] The means for obtaining human spleen and liver trauma ultrasound images are limited and the number is small. Some animals such as pigs have very close physiological functions, shapes and sizes to humans, and it is relatively easy to obtain animal spleen and liver trauma ultrasound images, so the animal spleen and liver trauma ultrasound images can be used for training of human clinical spleen and liver trauma automatic segmentation model, wherein the ultrasound image data set of animal spleen and liver trauma is obtained as a weak data set, and the human spleen and liver trauma ultrasound image data set is obtained as a strong data set.

[0050] S22, the convolutional neural network model and the data generation network are used to enhance the weak data set of animal spleen and liver trauma ultrasound images with the strong data set of human spleen and liver trauma ultrasound images, to obtain the data set generated after data enhancement of the weak data set of animal spleen and liver trauma ultrasound images, and to establish a pre-training data set, and the human spleen and liver trauma ultrasound image data set is used to establish a fine-tuning data set, wherein the pre-training data set and the fine-tuning data set both include normal non-trauma images and trauma images.

[0051] As Figure 1As shown, the convolutional neural network model learns the distribution rules of different training data sets within the same class from the feature distribution of the strong data set, extracts image features, structural features and semantic features, and uses the feature distribution rules contained in the strong data set of human spleen and liver trauma ultrasound images to enhance the weak data set of animal spleen and liver trauma ultrasound images through the data generation network, thereby expanding the training data in the weak data set of animal spleen and liver trauma ultrasound images, generating a human spleen and liver trauma ultrasound image generation data set, and improving the utilization rate of the weak data set of animal spleen and liver trauma ultrasound images.

[0052] The data enhancement model based on the convolutional neural network extracts features from the strong data set, and the weak data set learns the characteristics of the strong data set. Specifically, the animal spleen and liver trauma ultrasound images (weak data set) and the human spleen and liver trauma ultrasound images (strong data set) are put into the data enhancement model of Figure 2 , and the data enhancement model is trained to learn the differentiated features of the strong data set based on the weak data set. Then all the weak data sets are put into the trained data enhancement model, and the pre-trained data set after data enhancement is output. The training data set of the data enhancement model is the animal spleen and liver trauma ultrasound images (weak data set) and the human spleen and liver trauma ultrasound images (strong data set), and the network structure of the data enhancement model is designed as follows: ResNet residual network structure is adopted, and the output of each residual network is the input of the next stage through the down-sampling coding of two residual network blocks; then the convolution layer, the BatchNorm normalization layer and the ReLU activation function are sequentially passed through to obtain the enhanced feature map, as shown in Figure 2 .

[0053] The data enhancement model trained by the strong data set processes the weak data set, so that the animal spleen and liver trauma ultrasound image data set can be closer to the human spleen and liver trauma ultrasound image data set in image features, structural features and semantic features, thereby obtaining a larger spleen and liver trauma ultrasound image generation data set.

[0054] After the above data set expansion, the data set is arranged and divided: the data set generated by the weak data set of about 6000 animal spleen and liver trauma ultrasound images after data enhancement obtained by the above steps is used to establish the model pre-training data set, and about 800 human spleen and liver trauma ultrasound image data sets are used to establish the fine-tuning data set, which includes normal non-trauma images and trauma images.

[0055] S23, manually label the pre-training data set and the fine-tuning data set respectively to form corresponding manually labeled segmentation label data sets and trauma grade classification label data sets, the labels of the manually labeled segmentation label data set include background, spleen / liver structure, and spleen / liver trauma, and the trauma grade in the trauma grade classification label data set is divided into 1-5 according to the degree.

[0056] Further, the segmentation label images in the artificially annotated segmentation label dataset are converted into multi-channel images using one-hot encoding, each channel representing a class, and each target structure occupying one channel of the label image, the target structure being background, organ or trauma, i.e. the artificially annotated segmentation labels of RGB or grayscale type spleen and liver trauma ultrasound images are converted into three-channel images, the 0th channel representing the background, the 1st channel representing the spleen / liver range, and the 2nd channel representing the trauma range. As shown in Figure 3 .

[0057] S24, based on the pre-training dataset, the fine-tuning dataset and the artificially annotated segmentation label dataset, an automatic segmentation model for spleen and liver trauma ultrasound images is established, and the automatic segmentation model for spleen and liver trauma ultrasound images is used for clinical spleen and liver trauma ultrasound image processing to obtain segmentation prediction results, including spleen and liver positions and contours, and injured parts and / or injured contour information.

[0058] The establishment of the automatic segmentation model for spleen and liver trauma ultrasound images is based on the pre-training dataset, the fine-tuning dataset and the artificially annotated segmentation label dataset generated and divided in step S22. First, the pre-training dataset and the corresponding artificially annotated segmentation label dataset are used for pre-training. Considering the slight differences between the generated data and the clinical images, after obtaining the segmentation model through pre-training, the fine-tuning dataset and the corresponding artificially annotated segmentation label dataset are used for fine-tuning, and finally the automatic segmentation model for spleen and liver trauma ultrasound images is obtained. Therefore, in the training scheme, the pre-training dataset and the corresponding artificially annotated segmentation label dataset are used for pre-training for 400 epochs, and then the fine-tuning dataset and the corresponding artificially annotated segmentation label dataset are used for fine-tuning for 100 epochs, and finally the automatic segmentation model for spleen and liver trauma ultrasound images is established.

[0059] The automatic segmentation model for spleen and liver trauma ultrasound images adopts an optimized design based on local context information and multi-scale context information based on the Transformer architecture, and the specific network structure is as shown in Figure 4 .

[0060] (1) The input video image frame sequence is enlarged or reduced through the image scaling module, preferably bilinear interpolation, and then passed through the convolution layer to make the size and dimension of the feature map of the corresponding decoding stage the same as the size and dimension of the feature map.

[0061] (2) Supervision and loss calculation of the encoding stage (encoder), which includes four stages, each of which includes pixel grouping / downsampling processing of input features and a Transformer block. The input of each stage is the downsampled output of the previous stage, and the output of each stage is saved. The Transformer block includes a self-attention calculation module, a first normalization layer, a full connection layer, and a second normalization layer, wherein the full connection layer is replaced by a channel block local context information extraction module, and the structure of the Transformer block is as shown in Figure 5 The processing of the channel block local context information extraction module on the input features specifically includes:

[0062] ① Use a 1x1 convolution to compress the input feature F in to obtain a feature map F0.

[0063] ② Split F0 into s block structures with the same dimension according to the channel dimension, and the dimension of each block structure is The output y1 of the first branch is the first block structure F1.

[0064] ③ Use a 3x3 depth separable convolution DWConv on the remaining branches to extract features on the input features in parallel, and concatenate between the branches except the first and second branches, so that the output of each branch is not only based on the block structure of F0 as input, but also obtains features of different scales in the previous branches. The output of each branch is y i , and the formula is as follows:

[0065]

[0066] ④ Merge the feature maps of each branch to obtain fused local information, and use a 1x1 convolution conv to compress the channel number again, and perform residual calculation with the input feature of the module to obtain the output feature map.

[0067] F = conv(concat[y i ,...,y s ]) (2)

[0068]

[0069] (2) Supervision and loss calculation of the decoding stage (decoder), which includes four stages of decoding, each of which includes a Decoder block. The output of the fourth encoding stage is used as the input of the first stage, and the upsampled output of the previous decoding stage and the output of the corresponding third, second, and first encoding stages are used as the input of the second, third, and fourth stages, respectively. The input is Each Decoder block includes a convolutional layer, a BatchNorm normalization layer and a ReLU activation function, and after outputting the feature maps of each channel in the fourth stage, the final result is convolved to obtain the segmentation prediction result.

[0070] S25, based on the pre-training data set, the fine-tuning data set and the corresponding artificial annotation of the trauma grade classification label data set, training is carried out to establish a spleen and liver trauma ultrasound image grade classification model, and the segmentation prediction result output in step S24 is predicted using the spleen and liver trauma ultrasound image grade classification model to obtain the corresponding spleen and liver trauma grade classification label.

[0071] Among them, the establishment of the spleen and liver trauma ultrasound image grade classification model is based on the pre-training data set, the fine-tuning data set and the corresponding artificial annotation of the trauma grade classification label data set mentioned in steps S22 and S23 for training. First, the pre-training data set and the corresponding artificial annotation of the trauma grade classification label data set are used for pre-training, then the fine-tuning data set and the corresponding artificial annotation of the trauma grade classification label data set are used for fine-tuning, and finally the spleen and liver trauma ultrasound image grade classification model is established. The specific network structure design of the spleen and liver trauma ultrasound image grade classification model is as follows: the input data set passes through three convolutional blocks Conv2D, each of which has a maximum pooling layer MaxPooling2D, and after the three convolutional blocks, a fully connected layer and a Softmax classifier are used to classify the extracted features to obtain the spleen and liver trauma grade classification result of the input image. As shown in Figure 6 .

[0072] The scheme of the application can also be replaced as follows:

[0073] 1. Adjustment of the model training scheme based on data enhancement and generated data set

[0074] The training scheme based on data enhancement and generated data set is adjusted, for example, a generated data model with a different network structure is used, or a different model training method including online transfer learning is used, or the distribution and proportion of animal and clinical human training data sets in the application are adjusted, which can be used as an alternative scheme for completing the model training in the application.

[0075] 2. Adjustment of the spleen and liver trauma ultrasound image automatic segmentation model

[0076] The spleen and liver trauma ultrasound image automatic segmentation model in the application is based on the advantage of capturing global context information by the Transformer, and extracts local context information and multi-scale context information in the spleen and liver trauma ultrasound image. The extraction scheme of the local context information and the multi-scale context information is adjusted, for example, other convolution structures based on a hollow convolution are used for local context information extraction, or a cross-attention mechanism is used for multi-scale context information extraction, which can be used as an alternative scheme to complete the spleen and liver trauma ultrasound image automatic segmentation model in the application.

[0077] 3. Adjustment of the structure of the spleen and liver trauma ultrasound image grade classification model

[0078] The network structure of the spleen and liver trauma ultrasound image grade classification model is adjusted, for example, other CNN multi-classification network structures, which can be used as an alternative scheme for the classification prediction process in the spleen trauma ultrasound image segmentation and classification system.

[0079] The application adopts a data enhancement and generation method to train the spleen and liver trauma ultrasound image automatic segmentation model and the spleen and liver trauma ultrasound image grade classification model. First, the ultrasound images of the spleen and liver trauma animals are combined through a convolutional neural network with residual learning, and data enhancement is used to obtain a strong data set closer to clinical data, and a larger scale of spleen and liver trauma ultrasound image data set is obtained; then the animal spleen and liver trauma ultrasound image data set and the corresponding artificial annotation segmentation label data set and grade classification label data set are used for pre-training, and the clinical spleen and liver trauma ultrasound image data set and the corresponding artificial annotation segmentation label data set and grade classification label data set are used for fine-tuning, and finally an artificial intelligence assisted ultrasound diagnosis spleen and liver trauma model is established.

[0080] The role of data enhancement and generation is to learn the distribution rules of different training data sets in the same class from the feature distribution of the strong training data set, extract image features, structure features and semantic features, and use the feature distribution rules contained in the strong data set to enhance the weak training data set through the data generation network, so as to expand the training data in the weak data set and improve the utilization rate of the weak data set in training.

[0081] The ultrasound diagnosis spleen trauma segmentation model adopted in the application is optimized in terms of local context information and multi-scale context information.

[0082] On the basis of the advantage of capturing global context information by using the Transformer, the local context information in the spleen and liver trauma ultrasound image is extracted. Therefore, the channel block local context information extraction module is used in the spleen and liver trauma ultrasound image automatic segmentation model in the application to replace the traditional FFN module, so that the problem of ignoring the local context information of details caused by the global matrix operation in the existing Transformer framework is solved. The local context information is obtained by the channel block local context information extraction module to supplement the semantic information in the small range area, so that the segmentation performance on the clinical spleen and liver trauma ultrasound image is improved, and the image details with high position sensitivity and fine granularity are mined.

[0083] On the basis of the advantage of capturing global context information by using the Transformer, the multi-scale context information in the spleen and liver trauma ultrasound image is extracted. Therefore, the U-Net-based skip connection is used in the spleen and liver trauma ultrasound image automatic segmentation model in the application on the basis of the existing Transformer framework, that is, the encoder output is transmitted to the corresponding level decoder as the input and is spliced with the up-sampling result of the previous layer decoder, so that the spatial information loss caused by the down-sampling process is reduced, the feature map recovered by the up-sampling contains more low-level semantic information through the splicing of the feature map in the channel dimension, which is helpful to recover the lesion details and the spatial dimension for accurate positioning, so that the fusion result across the resolution is obtained, and the representability of the multi-scale is improved.

[0084] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled persons in the technical field, several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be regarded as the protection scope of the present application.

Claims

1. A method for establishing an artificial intelligence-assisted ultrasound diagnosis model for spleen and liver trauma, characterized in that: The method comprises the following steps: S21, acquiring an ultrasound image data set of animal spleen and liver trauma as a weak data set, and acquiring a human spleen and liver trauma ultrasound image data set as a strong data set; S22, using a convolutional neural network model and a data generation network to enhance the weak data set of the animal spleen and liver trauma ultrasound image with the strong data set of the human spleen and liver trauma ultrasound image, to obtain a data set generated after data enhancement of the weak data set of the animal spleen and liver trauma ultrasound image, to establish a pre-training data set, and using the human spleen and liver trauma ultrasound image data set to establish a fine-tuning data set, wherein the pre-training data set and the fine-tuning data set each comprise normal non-trauma images and trauma images; S23, manually labeling the pre-training data set and the fine-tuning data set respectively to form corresponding manually labeled segmentation label data sets and trauma grade classification label data sets; wherein the labels of the manually labeled segmentation label data set comprise background, spleen / liver structure, and spleen / liver trauma; S24, establishing a spleen and liver trauma ultrasound image automatic segmentation model based on the pre-training data set, the fine-tuning data set, and the manually labeled segmentation label data set, and using the spleen and liver trauma ultrasound image automatic segmentation model to process clinical spleen and liver trauma ultrasound images to obtain segmentation prediction results, wherein the segmentation prediction results comprise spleen and liver positions and spleen and liver contours, and injured parts and / or injured contour information; S25, training a spleen and liver trauma ultrasound image grade classification model based on the pre-training data set, the fine-tuning data set, and the corresponding manually labeled trauma grade classification label data set, and using the spleen and liver trauma ultrasound image grade classification model to predict the segmentation prediction results output in step S24 to obtain corresponding spleen and liver trauma grade classification labels; The network structure of the spleen and liver trauma ultrasound image automatic segmentation model comprises an image scaling module, a convolutional layer, an encoding stage, and a decoding stage; The input video image frame sequence is enlarged or reduced through the image scaling module, and then passes through the convolutional layer, so that the size and dimension of the sequence are the same as the size and dimension of the feature map of the corresponding decoding stage; Supervision and loss calculation are performed in the encoding stage, which comprises four stages, each stage comprising pixel grouping / downsampling processing of input features and a Transformer block, the input of each stage being the downsampled output of the previous stage, and the output of each stage being saved; the Transformer block comprises a self-attention calculation module, a first normalization layer, a channel block local context information extraction module, and a second normalization layer; Supervision and loss calculation are performed in the decoding stage, which comprises four stages of decoding, each stage comprising a Decoder block; wherein the output of the fourth encoding stage is used as the input of the first stage, the upsampled output of the previous decoding stage and the output of the corresponding third, second, and first encoding stages are used as the inputs of the second, third, and fourth stages respectively; each Decoder block comprises a convolutional layer, a BatchNorm normalization layer, and a ReLU activation function, and after outputting the feature maps of each channel in the fourth stage, the final result is convolved to obtain the segmentation prediction result. ​ The processing of the input features by the channel block local context information extraction module specifically includes: compressing input features using 1x1 convolution number of channels, resulting in a feature map ; Will Split into the same dimension according to the channel dimension The dimension of each block structure is The output of the first branch That is, the first block structure ; Using 3x3 depthwise separable convolution on the rest of the branches Parallelly extracting features on input features, concatenating between branches except the first and second branches, so that the output of each branch is not only based on The block structure of the input is as follows: the features of different scales in the previous branches are obtained at the same time; the output of each branch is The formula is as follows: （1） Merge each branch feature map to get the fused local information, and use 1x1 convolution again The number of compression channels is calculated with the residual of the input features of the module to obtain the output feature map. （2） （3）。 2. The method for establishing an artificial intelligence-assisted ultrasound diagnosis model of spleen and liver trauma according to claim 1, wherein: In the step S22, the convolutional neural network model learns the distribution rules of different training data sets in the same class from the feature distribution of the strong data set, extracts image features, structural features and semantic features, and uses the feature distribution rules contained in the strong data set of the human spleen and liver trauma ultrasound image to generate a weak data set of the animal spleen and liver trauma ultrasound image through the data generation network, thereby expanding the training data in the weak data set of the animal spleen and liver trauma ultrasound image and generating a human spleen and liver trauma ultrasound image generation data set.

3. The method of claim 1, wherein the model is established using artificial intelligence assisted ultrasound diagnosis of a splenic and hepatic trauma. In the step S22, the feature extraction is performed on the strong data set by the data enhancement model based on the convolutional neural network, and the weak data set is learned to obtain the characteristics of the strong data set; Specifically, the weak data set and the strong data set are put into the data enhancement model for training, so that the data enhancement model learns the differentiated features of the strong data set based on the weak data set, and then all the weak data sets are put into the trained data enhancement model to output the pre-training data set after data enhancement; the network structure of the data enhancement model is designed as follows: a ResNet residual network structure is adopted, and the output of each residual network is used as the input of the next stage through the down-sampling coding of two residual network blocks; then, a convolution layer, a BatchNorm normalization layer and a ReLU activation function are sequentially used to obtain the strengthened feature map.

4. The method of establishing an artificial intelligence assisted ultrasound diagnosis of a splenic and hepatic trauma model according to any one of claims 1 to 3, characterized in that, In the step S23, the segmentation label images in the artificially annotated segmentation label data set are converted into multi-channel images using one-hot encoding, each channel representing a class, and each target structure occupying a channel of the label image, the target structure being background, organ or trauma, that is, the artificially annotated segmentation label of the RGB or grayscale type spleen and liver trauma ultrasound image is converted into a three-channel image, the 0 channel representing background, the 1 channel representing the spleen / liver range, and the 2 channel representing the trauma range.

5. The method of establishing an artificial intelligence assisted ultrasound diagnosis of a splenic and hepatic trauma model according to any one of claims 1-3, wherein, In the step S24, the establishment of the spleen and liver trauma ultrasound image automatic segmentation model based on the pre-training data set, the fine-tuning data set and the artificially annotated segmentation label data set includes: first, pre-training is performed using the pre-training data set and the corresponding artificially annotated segmentation label data set, then fine-tuning is performed using the fine-tuning data set and the corresponding artificially annotated segmentation label data set, and finally the spleen and liver trauma ultrasound image automatic segmentation model is obtained.

6. The method of establishing an artificial intelligence assisted ultrasound diagnosis of a splenic liver trauma model of claim 5, wherein, The image scaling module adopts a bilinear difference method.

7. The method of establishing an artificial intelligence assisted ultrasound diagnosis of a splenic and hepatic trauma model of claim 6, wherein, In the step S25, the training of the spleen and liver trauma ultrasound image grade classification model based on the pre-training data set, the fine-tuning data set and the corresponding artificially annotated trauma grade classification label data set specifically includes: first, pre-training is performed using the pre-training data set and the corresponding artificially annotated trauma grade classification label data set, then fine-tuning is performed using the fine-tuning data set and the corresponding artificially annotated trauma grade classification label data set, and finally the spleen and liver trauma ultrasound image grade classification model is established.

8. The method for establishing an artificial intelligence-assisted ultrasound diagnosis model for spleen and liver trauma according to claim 7, wherein: The network structure of the spleen and liver trauma ultrasound image grade classification model is as follows: the input data set passes through three convolution blocks Conv2D, each of which has a maximum pooling layer MaxPooling2D, and after the three convolution blocks, a full connection layer and a Softmax classifier are used to classify the extracted features, so as to obtain the spleen and liver trauma grade classification result of the input image.

Images