Method and device for detecting phlebolithiasis based on deep learning

Abstract

The present invention provides a method and device for detecting phleboliths based on deep learning, the method comprising: obtaining an image sample with a phlebolith area marked; inputting the image sample into a preset deep learning network model for training to obtain an optimal training weight, and obtaining a phlebolith image detection network model based on the optimal training weight and the deep learning network model; inputting an ultrasound image to be identified into the phlebolith image detection network model to obtain an image marked with phlebolith information; wherein the phlebolith image detection network model comprises: a trunk feature extraction part, an enhanced feature fusion part, and a target detection output part. The present invention can solve the technical problem that phleboliths are difficult to identify in venous malformations.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for detecting phlebolithiasis based on deep learning. Background Art

[0002] Venous malformations are generally identified by grayscale ultrasound images of the target object and color blood flow ultrasound images before and after compression. However, the detection and identification are difficult due to factors such as difficult to distinguish ultrasound image features, rich texture, wide area of ​​involvement, blurred boundaries, and artifacts often generated in color blood flow ultrasound images, which affect the diagnosis results. Phleboliths are important diagnostic features in grayscale ultrasound images of venous malformations. Diagnosing phleboliths from grayscale ultrasound images of venous malformations can provide a reliable diagnostic basis for distinguishing venous malformations. Phleboliths often appear in venous malformations with low blood flow rates such as the extremities. In ultrasound images, there are difficulties in identification such as similar ultrasound features to the background, small targets, and possible manual misses. Clinical identification is difficult. Summary of the invention

[0003] In view of this, it is necessary to provide a phlebolith detection method and device based on deep learning to solve the technical problem that phleboliths are difficult to identify in venous malformations.

[0004] In order to achieve the above object, the present invention provides a method for detecting phleboliths based on deep learning, comprising:

[0005] Obtain image samples with areas labeled with phleboliths;

[0006] Inputting the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtaining a phlebolith image detection network model based on the optimal training weights and the deep learning network model;

[0007] Inputting the ultrasound image to be identified into the phlebolith image detection network model to obtain an image marked with phlebolith information;

[0008] The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part;

[0009] The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid;

[0010] The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path;

[0011] The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence;

[0012] The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence;

[0013] The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path.

[0014] Furthermore, the step of obtaining an image sample with a phlebolith area marked includes:

[0015] Acquiring grayscale ultrasound images of venous malformations, classifying and screening the grayscale ultrasound images of venous malformations, and obtaining grayscale ultrasound images containing phleboliths;

[0016] The grayscale ultrasound image containing phleboliths is preprocessed to obtain an image sample of the annotated phlebolith area.

[0017] Furthermore, the grayscale ultrasound image containing phleboliths is preprocessed to obtain an image sample of the annotated phlebolith area, including:

[0018] Preprocessing the grayscale ultrasound image containing phleboliths to hide personal identity information in the grayscale ultrasound image containing phleboliths to obtain a desensitized image;

[0019] Cropping the desensitized image to obtain a grayscale ultrasound image imaging area image;

[0020] The grayscale ultrasound image imaging area image is labeled to obtain an image sample of the labeled phlebolith area.

[0021] Furthermore, the activation function of the backbone concentrated convolution block in the backbone feature extraction part is SiLU.

[0022] Further, the backbone centralized convolution block includes: a first convolution batch normalization layer, a second convolution batch normalization layer, a third convolution batch normalization layer, a backbone bottleneck block, and a first splicing layer;

[0023] The first convolution batch normalization layer, the backbone bottleneck block, the first splicing layer, and the second convolution batch normalization layer are connected in sequence, and the output end of the third convolution batch normalization layer is connected to the input end of the first splicing layer;

[0024] The backbone bottleneck block includes a fourth convolutional batch normalization layer and a fifth convolutional batch normalization layer, the output end of the fourth convolutional batch normalization layer is connected to the input end of the fifth convolutional batch normalization layer, and the input end of the fourth convolutional batch normalization layer is also additively connected to the output end of the fifth convolutional batch normalization layer.

[0025] Furthermore, the fast feature pyramid module includes: a sixth convolutional batch normalization layer, a seventh convolutional batch normalization layer, a first maximum pooling layer, a second maximum pooling layer, a third maximum pooling layer and a second splicing layer;

[0026] Among them, the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are connected in sequence, the sixth convolutional batch normalization layer, the second splicing layer and the seventh convolutional batch normalization layer are connected in sequence, the sixth convolutional batch normalization layer is also connected to the first maximum pooling layer, and the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are also connected to the second splicing layer.

[0027] Further, the enhanced concentrated convolution block includes: an eighth convolution batch normalization layer, a ninth convolution batch normalization layer, a tenth convolution batch normalization layer, an enhanced bottleneck block and a third splicing layer;

[0028] Among them, the eighth convolutional batch normalization layer, the enhanced bottleneck block, the third splicing layer, and the ninth convolutional batch normalization layer are connected in sequence, and the output end of the tenth convolutional batch normalization layer is connected to the input end of the third splicing layer.

[0029] The present invention also provides a phlebolith detection device based on deep learning, comprising:

[0030] An acquisition module, used for acquiring image samples with phlebolith areas marked;

[0031] A training module, used for inputting the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtaining a phlebolith image detection network model based on the optimal training weights and the deep learning network model;

[0032] A recognition module, used for inputting the ultrasound image to be recognized into the phlebolith image detection network model to obtain an image marked with phlebolith information;

[0033] The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part;

[0034] The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid;

[0035] The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path;

[0036] The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence;

[0037] The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence;

[0038] The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path.

[0039] The present invention also provides an electronic device, comprising a memory and a processor, wherein:

[0040] The memory is used to store programs;

[0041] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the deep learning-based phlebolith detection method as described in any one of the above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the deep learning-based phlebolith detection method as described in any one of the above items is implemented.

[0043] The beneficial effect of adopting the above-mentioned implementation method is: the deep learning-based phlebolith detection method and device provided by the present invention obtains the optimal training weights by inputting image samples marked with phlebolith areas into a preset deep learning network model for training, and obtains the phlebolith image detection network model based on the optimal training weights and the deep learning network model, thereby detecting and identifying the ultrasound image to be identified.

[0044] The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part. The network model can realize automatic detection and identification of phleboliths to achieve the purpose of distinguishing venous malformations. Diagnosing venous malformations through phleboliths can provide a reliable diagnostic basis for distinguishing venous malformations and improve the accuracy of phlebolith recognition and venous malformation identification.

[0045] In the backbone feature extraction part of the network model, the training function and structure are adjusted, and the multi-path residual structure is used to splice and fuse features of different scales, which enhances the effect of feature extraction. In the enhanced feature fusion part of the network model, two feature fusion paths, bottom-up and top-down, are used. Bottom-up feature fusion helps convey strong feature positioning information, and top-down helps convey strong feature semantic information, which enhances the effect of feature fusion.

[0046] Residual structures and multi-scale convolution operations of different scales are added to the backbone feature extraction part and the enhanced feature fusion part of the network model to extract and fuse the features in the ultrasound image from the multi-scale space, thereby enhancing the network model's ability to extract local and overall features and the fusion effect.

[0047] By adding target detection output layers of different scales in the network model, targets are detected and identified at multiple scales, so that the training model can better fit targets of different sizes, improving the target detection effect, thereby solving the technical problem of difficult identification of phleboliths in venous malformations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of a process flow of an embodiment of a phlebolith detection method based on deep learning provided by the present invention;

[0050] Figure 2 A schematic diagram of a training deep learning network model provided by the present invention;

[0051] Figure 3 A schematic diagram of ultrasound image data preprocessing provided by the present invention;

[0052] Figure 4 A schematic diagram of a backbone centralized convolutional block provided by the present invention;

[0053] Figure 5A schematic diagram of a fast feature pyramid module provided by the present invention;

[0054] Figure 6 A schematic diagram of the enhanced centralized convolution block provided by the present invention;

[0055] Figure 7 A schematic flow chart of another embodiment of a phlebolith detection method based on deep learning provided by the present invention;

[0056] Figure 8 It is a principle block diagram of a phlebolith detection device based on deep learning provided by the present invention;

[0057] Fig. 9 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0059] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two.

[0060] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment comprising a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or equipment.

[0061] The naming or numbering of the steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0062] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] The present invention provides a method and device for detecting phlebolithiasis based on deep learning, which are described below respectively.

[0064] like Figure 1 As shown, the present invention provides a method for detecting phleboliths based on deep learning, comprising:

[0065] Step 110, obtaining an image sample with a phlebolith area marked;

[0066] Step 120: input the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtain a phlebolith image detection network model based on the optimal training weights and the deep learning network model;

[0067] Step 130, inputting the ultrasound image to be identified into the phlebolith image detection network model to obtain an image marked with phlebolith information;

[0068] Among them, Figure 2 As shown, the phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part;

[0069] The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid;

[0070] The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path;

[0071] The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence;

[0072] The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence;

[0073] The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path.

[0074] It can be understood that the image marked with phlebolith information obtained through the above steps can be further used to identify venous malformations.

[0075] The deep learning network model is constructed. The training and detection network models used in the present invention are consistent and are divided into three parts: a backbone feature extraction part, an enhanced feature fusion part, and a target detection output part.

[0076] The backbone feature extraction part includes residual structure and multi-scale convolution operations. First, the input image is processed preliminarily by one convolution, and then five convolutions are performed and input to the backbone centralized convolution block. The backbone centralized convolution block is a convolution module with the residual output removed. The activation function in the spliced ​​standard convolution module is changed from LeakyReLU to SiLU. The backbone bottleneck block in the backbone centralized convolution block reduces and then expands the number of channels (by half by default) to obtain features (a total of two standard convolution modules are used). The number of input and output channels does not change. Then it is input into the fast pyramid module, and the maximum pooling operation with different pooling kernel sizes is used to continue to extract features to improve the receptive field of the network. After completing the backbone feature extraction part, four preliminary feature images are obtained. All feature images are input into the enhanced feature fusion part, which includes two feature fusion paths, bottom-up and top-up.

[0077] First, a bottom-up feature fusion operation is performed, and the fourth feature image is subjected to multi-scale convolution and upsampling operations, and then spliced ​​with the third feature image and input into the enhanced concentrated convolution block of the enhanced feature fusion part. The difference between the enhanced concentrated convolution block and the trunk concentrated convolution block is that the trunk concentrated convolution block performs multiple feature layer addition operations, while the enhanced concentrated convolution block does not perform multiple feature layer addition operations. The bottom-up feature fusion performs three convolution, upsampling, splicing and input operations into the enhanced concentrated convolution block to convey strong positioning features.

[0078] Then, the top-down feature fusion path is performed, and the feature layer that has completed the bottom-up fusion is input into the top-down feature fusion part for convolution. After being spliced ​​with the feature layer of the same size in the bottom-up feature fusion path, it is input into the enhanced concentrated convolution block. A total of three operations of convolution, splicing and input into the enhanced concentrated convolution block are performed to convey strong semantic information.

[0079] Finally, the output layer of the last enhanced centralized convolution group in the bottom-up feature fusion and the output of each enhanced centralized convolution block in the top-down feature fusion are selected as the output layer of the target detection output part, and target detection is performed on outputs of different scales.

[0080] Finally, the recognition results in all scale spaces are fused and compared with the annotation information, and the training data is continuously updated to obtain the optimal training weights.

[0081] The original grayscale ultrasound images and annotation information of phleboliths were input into the constructed deep learning network model for training to obtain the optimal training weights.

[0082] The ultrasound images suspected of being venous malformations that need to be detected and identified are input into the phlebolith image detection network model, and the optimal training weights are used for detection and identification. The grayscale ultrasound images with the areas and names of phleboliths marked are output as an auxiliary basis for clinicians' diagnosis.

[0083] In some embodiments, obtaining an image sample with a phlebolith region marked includes:

[0084] Acquiring grayscale ultrasound images of venous malformations, classifying and screening the grayscale ultrasound images of venous malformations, and obtaining grayscale ultrasound images containing phleboliths;

[0085] The grayscale ultrasound image containing phleboliths is preprocessed to obtain an image sample of the annotated phlebolith area.

[0086] It is understandable that the grayscale ultrasound images of venous malformations may be of different types, some of which contain phleboliths and some of which do not. Therefore, the acquired grayscale ultrasound images of venous malformations need to be classified and screened.

[0087] Grayscale ultrasound images containing phleboliths are preprocessed to eliminate irrelevant information in the images, retain useful real information, enhance the detectability of relevant information and simplify the data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition.

[0088] In some embodiments, the preprocessing of the grayscale ultrasound image containing phleboliths to obtain an image sample of the annotated phlebolith area includes:

[0089] Preprocessing the grayscale ultrasound image containing phleboliths to hide personal identity information in the grayscale ultrasound image containing phleboliths to obtain a desensitized image;

[0090] Cropping the desensitized image to obtain a grayscale ultrasound image imaging area image;

[0091] The grayscale ultrasound image imaging area image is labeled to obtain an image sample of the labeled phlebolith area.

[0092] It is understandable that all grayscale ultrasound images are preprocessed, and after each person's personal information is hidden, the ultrasound images are cropped to retain only the grayscale ultrasound image imaging area, and the area containing phleboliths is labeled. The process of generating the final image sample, i.e., the labeled image, is as follows: Figure 3 shown.

[0093] In some embodiments, the activation function of the backbone concentrated convolution block in the backbone feature extraction part is SiLU.

[0094] It can be understood that the SiLU activation function is also a sigmoid weighted linear unit.

[0095] In some embodiments, Figure 4 As shown, the backbone centralized convolution block includes: a first convolution batch normalization layer, a second convolution batch normalization layer, a third convolution batch normalization layer, a backbone bottleneck block, and a first splicing layer;

[0096] The first convolution batch normalization layer, the backbone bottleneck block, the first splicing layer, and the second convolution batch normalization layer are connected in sequence, and the output end of the third convolution batch normalization layer is connected to the input end of the first splicing layer;

[0097] The backbone bottleneck block includes a fourth convolutional batch normalization layer and a fifth convolutional batch normalization layer, the output end of the fourth convolutional batch normalization layer is connected to the input end of the fifth convolutional batch normalization layer, and the input end of the fourth convolutional batch normalization layer is also additively connected to the output end of the fifth convolutional batch normalization layer.

[0098] In some embodiments, Figure 5 As shown, the fast feature pyramid module includes: a sixth convolution batch normalization layer, a seventh convolution batch normalization layer, a first maximum pooling layer, a second maximum pooling layer, a third maximum pooling layer and a second splicing layer;

[0099] Among them, the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are connected in sequence, the sixth convolutional batch normalization layer, the second splicing layer and the seventh convolutional batch normalization layer are connected in sequence, the sixth convolutional batch normalization layer is also connected to the first maximum pooling layer, and the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are also connected to the second splicing layer.

[0100] In some embodiments, Figure 6 As shown, the enhanced centralized convolution block includes: an eighth convolution batch standardization layer, a ninth convolution batch standardization layer, a tenth convolution batch standardization layer, an enhanced bottleneck block and a third splicing layer;

[0101] Among them, the eighth convolutional batch normalization layer, the enhanced bottleneck block, the third splicing layer, and the ninth convolutional batch normalization layer are connected in sequence, and the output end of the tenth convolutional batch normalization layer is connected to the input end of the third splicing layer.

[0102] In some embodiments, the process of the phlebolith detection method based on deep learning is as follows Figure 7As shown, after collecting the ultrasonic image data, preprocessing is performed and a module is constructed to perform detection and recognition.

[0103] In summary, the present invention proposes a method for detecting and identifying venous grayscale ultrasound images in venous malformations based on an improved yolov5 deep learning network model. First, from the collected patient ultrasound images, the venous grayscale ultrasound images for training are classified and screened, personal information is hidden from all grayscale ultrasound images, only the grayscale ultrasound image imaging area is retained by cropping, and all images are annotated; then the original ultrasound image and its annotated information are used as a set of training data, input into the training network model, and after downscaling feature extraction is performed in the trunk feature extraction part with residual structure multi-scale convolution, the extracted features are input into the enhanced feature fusion part with residual structure and multi-scale convolution for bottom-up and top-down feature fusion. Then, target feature layers of different scales are selected from the fused features as outputs for target detection and recognition. Finally, the recognition results in all scale spaces are fused, compared with the annotated information, and the training data is continuously updated to obtain the optimal training weights.

[0104] The grayscale ultrasound image of the vein to be detected is input into the detection network model, and the trained optimal weight is used for detection and recognition. The grayscale ultrasound image with the area and name of the phlebolith marked is output as an auxiliary basis for clinical diagnosis. The experimental results show that the accuracy of this method in identifying phleboliths reaches 94.7%, the recall rate reaches 89.5%, and the mAP val 50 Reaching 94.3%, mAP val 50-95 Reached 52.9%.

[0105] The deep learning-based phlebolith detection method provided by the present invention obtains optimal training weights by inputting image samples marked with phlebolith areas into a preset deep learning network model for training, and obtains a phlebolith image detection network model based on the optimal training weights and the deep learning network model, thereby detecting and identifying the ultrasound image to be identified.

[0106] The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part. The network model can realize automatic detection and identification of phleboliths to achieve the purpose of distinguishing venous malformations. Diagnosing venous malformations through phleboliths can provide a reliable diagnostic basis for distinguishing venous malformations and improve the accuracy of phlebolith recognition and venous malformation identification.

[0107] In the backbone feature extraction part of the network model, the training function and structure are adjusted, and the multi-path residual structure is used to splice and fuse features of different scales, which enhances the effect of feature extraction. In the enhanced feature fusion part of the network model, two feature fusion paths, bottom-up and top-down, are used. Bottom-up feature fusion helps convey strong feature positioning information, and top-down helps convey strong feature semantic information, which enhances the effect of feature fusion.

[0108] Residual structures and multi-scale convolution operations of different scales are added to the backbone feature extraction part and the enhanced feature fusion part of the network model to extract and fuse the features in the ultrasound image from the multi-scale space, thereby enhancing the network model's ability to extract local and overall features and the fusion effect.

[0109] By adding target detection output layers of different scales in the network model, targets are detected and identified at multiple scales, so that the training model can better fit targets of different sizes, improving the target detection effect, thereby solving the technical problem of difficult identification of phleboliths in venous malformations. Venous malformations can be identified by automatically detecting and identifying phleboliths.

[0110] like Figure 8 As shown, the present invention also provides a phlebolith detection device 800 based on deep learning, comprising:

[0111] An acquisition module 810 is used to acquire an image sample with a region marked with phleboliths;

[0112] A training module 820 is used to input the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtain a phlebolith image detection network model based on the optimal training weights and the deep learning network model;

[0113] The recognition module 830 is used to input the ultrasound image to be recognized into the phlebolith image detection network model to obtain an image marked with phlebolith information;

[0114] The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part;

[0115] The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid;

[0116] The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path;

[0117] The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence;

[0118] The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence;

[0119] The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path.

[0120] The deep learning-based phlebolith detection device provided in the above-mentioned embodiment can implement the technical solution described in the above-mentioned deep learning-based phlebolith detection method embodiment. The specific implementation principles of the above-mentioned modules or units can refer to the corresponding contents in the above-mentioned deep learning-based phlebolith detection method embodiment, which will not be repeated here.

[0121] like Fig. 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902 and a display 903. Fig. 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0122] In some embodiments, the memory 902 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 902 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 900.

[0123] Furthermore, the memory 902 may include both an internal storage unit of the electronic device 900 and an external storage device. The memory 902 is used to store application software installed in the electronic device 900 and various data.

[0124] In some embodiments, the processor 901 can be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 902, such as the deep learning-based phlebolith detection method of the present invention.

[0125] In some embodiments, the display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 903 is used to display information on the electronic device 900 and to display a visual user interface. The components 901-903 of the electronic device 900 communicate with each other via a system bus.

[0126] In some embodiments of the present invention, when the processor 901 executes the deep learning-based phlebolith detection method program in the memory 902, the following steps may be implemented:

[0127] Obtain image samples with areas labeled with phleboliths;

[0128] Inputting the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtaining a phlebolith image detection network model based on the optimal training weights and the deep learning network model;

[0129] Inputting the ultrasound image to be identified into the phlebolith image detection network model to obtain an image marked with phlebolith information;

[0130] The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part;

[0131] The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid;

[0132] The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path;

[0133] The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence;

[0134] The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence;

[0135] The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path.

[0136] It should be understood that when the processor 901 executes the deep learning-based venous stone detection method program in the memory 902, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0137] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 900 mentioned, and the electronic device 900 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0138] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the deep learning-based phlebolith detection method provided by the above methods, the method comprising:

[0139] Obtain image samples with areas labeled with phleboliths;

[0140] Inputting the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtaining a phlebolith image detection network model based on the optimal training weights and the deep learning network model;

[0141] Inputting the ultrasound image to be identified into the phlebolith image detection network model to obtain an image marked with phlebolith information;

[0142] The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part;

[0143] The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid;

[0144] The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path;

[0145] The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence;

[0146] The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence;

[0147] The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path.

[0148] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0149] The above is a detailed introduction to the deep learning-based venous stone detection method and device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting phlebolithiasis based on deep learning, characterized in that: include: Obtain image samples with areas labeled with phleboliths; Inputting the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtaining a phlebolith image detection network model based on the optimal training weights and the deep learning network model; Inputting the ultrasound image to be identified into the phlebolith image detection network model to obtain an image marked with phlebolith information; The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part; The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid; The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path; The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence; The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence; The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path; The backbone centralized convolution block includes: a first convolution batch normalization layer, a second convolution batch normalization layer, a third convolution batch normalization layer, a backbone bottleneck block, and a first splicing layer; The first convolution batch normalization layer, the backbone bottleneck block, the first splicing layer, and the second convolution batch normalization layer are connected in sequence, and the output end of the third convolution batch normalization layer is connected to the input end of the first splicing layer; The backbone bottleneck block includes a fourth convolutional batch normalization layer and a fifth convolutional batch normalization layer, the output end of the fourth convolutional batch normalization layer is connected to the input end of the fifth convolutional batch normalization layer, and the input end of the fourth convolutional batch normalization layer is also additively connected to the output end of the fifth convolutional batch normalization layer; The fast feature pyramid module includes: a sixth convolution batch normalization layer, a seventh convolution batch normalization layer, a first maximum pooling layer, a second maximum pooling layer, a third maximum pooling layer and a second splicing layer; Among them, the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are connected in sequence, the sixth convolutional batch normalization layer, the second splicing layer and the seventh convolutional batch normalization layer are connected in sequence, the sixth convolutional batch normalization layer is also connected to the first maximum pooling layer, and the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are also connected to the second splicing layer; The enhanced concentrated convolution block includes: an eighth convolution batch normalization layer, a ninth convolution batch normalization layer, a tenth convolution batch normalization layer, an enhanced bottleneck block and a third splicing layer; Among them, the eighth convolutional batch normalization layer, the enhanced bottleneck block, the third splicing layer, and the ninth convolutional batch normalization layer are connected in sequence, and the output end of the tenth convolutional batch normalization layer is connected to the input end of the third splicing layer.

2. The method for detecting phleboliths based on deep learning according to claim 1, characterized in that: The step of obtaining an image sample with a phlebolith area marked includes: Acquiring grayscale ultrasound images of venous malformations, classifying and screening the grayscale ultrasound images of venous malformations, and obtaining grayscale ultrasound images containing phleboliths; The grayscale ultrasound image containing phleboliths is preprocessed to obtain an image sample of the annotated phlebolith area.

3. The method for detecting phleboliths based on deep learning according to claim 2, characterized in that: The preprocessing of the grayscale ultrasound image containing phleboliths to obtain an image sample of the annotated phlebolith area includes: Preprocessing the grayscale ultrasound image containing phleboliths to hide personal identity information in the grayscale ultrasound image containing phleboliths to obtain a desensitized image; Cropping the desensitized image to obtain a grayscale ultrasound image imaging area image; The grayscale ultrasound image imaging area image is labeled to obtain an image sample of the labeled phlebolith area.

4. The method for detecting phleboliths based on deep learning according to claim 1, characterized in that: The activation function of the backbone concentrated convolution block in the backbone feature extraction part is SiLU.

5. A phlebolith detection device based on deep learning, characterized in that: include: An acquisition module, used for acquiring image samples with phlebolith areas marked; A training module, used for inputting the image samples into a preset deep learning network model for training to obtain optimal training weights, and obtaining a phlebolith image detection network model based on the optimal training weights and the deep learning network model; A recognition module, used for inputting the ultrasound image to be recognized into the phlebolith image detection network model to obtain an image marked with phlebolith information; The phlebolith image detection network model includes: a trunk feature extraction part, an enhanced feature fusion part and a target detection output part; The backbone feature extraction part includes a convolution layer for preliminary processing of the input image and 5 groups of convolution layers and backbone concentrated convolution blocks connected in sequence, and the backbone concentrated convolution block at the bottom layer is connected to a fast feature pyramid; The enhanced feature fusion part includes a bottom-up feature fusion path and a corresponding top-down feature fusion path; The bottom-up feature fusion path includes three first fusion modules connected in sequence, wherein the first fusion module includes a convolution layer, an upsampling layer, a concatenation layer, and an enhanced centralized convolution block connected in sequence; The top-down feature fusion path includes three second fusion modules connected in sequence, and the second fusion module includes a convolution layer, a splicing layer and an enhanced concentrated convolution block connected in sequence; The target detection part includes four convolutional layers, and the four convolutional layers are respectively connected to the topmost enhanced concentrated convolutional block of the bottom-up feature fusion path and the three enhanced concentrated convolutional blocks of the top-down feature fusion path; The backbone centralized convolution block includes: a first convolution batch normalization layer, a second convolution batch normalization layer, a third convolution batch normalization layer, a backbone bottleneck block, and a first splicing layer; The first convolution batch normalization layer, the backbone bottleneck block, the first splicing layer, and the second convolution batch normalization layer are connected in sequence, and the output end of the third convolution batch normalization layer is connected to the input end of the first splicing layer; The backbone bottleneck block includes a fourth convolutional batch normalization layer and a fifth convolutional batch normalization layer, the output end of the fourth convolutional batch normalization layer is connected to the input end of the fifth convolutional batch normalization layer, and the input end of the fourth convolutional batch normalization layer is also additively connected to the output end of the fifth convolutional batch normalization layer; The fast feature pyramid module includes: a sixth convolution batch normalization layer, a seventh convolution batch normalization layer, a first maximum pooling layer, a second maximum pooling layer, a third maximum pooling layer and a second splicing layer; Among them, the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are connected in sequence, the sixth convolutional batch normalization layer, the second splicing layer and the seventh convolutional batch normalization layer are connected in sequence, the sixth convolutional batch normalization layer is also connected to the first maximum pooling layer, and the first maximum pooling layer, the second maximum pooling layer and the third maximum pooling layer are also connected to the second splicing layer; The enhanced concentrated convolution block includes: an eighth convolution batch normalization layer, a ninth convolution batch normalization layer, a tenth convolution batch normalization layer, an enhanced bottleneck block and a third splicing layer; Among them, the eighth convolutional batch normalization layer, the enhanced bottleneck block, the third splicing layer, and the ninth convolutional batch normalization layer are connected in sequence, and the output end of the tenth convolutional batch normalization layer is connected to the input end of the third splicing layer.

6. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the deep learning-based venous stone detection method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the deep learning-based phlebolith detection method according to any one of claims 1 to 4 is implemented.

Images