Ultrasonic image processing method, device, equipment and storage medium

Abstract

The present application discloses an ultrasonic image processing method, device, equipment and storage medium. In the scheme, the ultrasonic image is automatically evaluated using a pre-trained deep learning neural network model to obtain a section evaluation result of the ultrasonic image; if the section evaluation result meets the preset requirements, it indicates that the sagittal section of the fetal sacral and coccygeal spine in the ultrasonic image is clear enough, and the spinal conus medullaris can be clearly seen with a high probability. Therefore, based on the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae in the ultrasonic image identified by the deep learning neural network model, the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae are marked in the ultrasonic image to obtain a marked ultrasonic image, which can help doctors assist in determining the position of the spinal conus medullaris terminal, thereby improving the accuracy and efficiency of the examination. Correspondingly, the ultrasonic image processing device, equipment and storage medium provided by the present application also have the above-mentioned technical effects.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an ultrasonic image processing method, device, equipment and storage medium. Background Art

[0002] During fetal ultrasound examination, doctors can manually intercept the sagittal section of the sacral and coccygeal spine in the ultrasound image and visually check the position of the conus medullaris on this section. However, since there are many vertebrae on this section (usually about a dozen vertebrae) and they are densely packed, visual inspection takes a long time and is prone to errors. Summary of the invention

[0003] In view of this, the purpose of this application is to provide an ultrasound image processing method, device, equipment and storage medium to automatically identify and mark the end of the conus medullaris, sacral vertebrae and lumbar vertebrae in ultrasound images to improve the accuracy and efficiency of the examination. The specific scheme is as follows:

[0004] To achieve the above objectives, on the one hand, the present application provides an ultrasound image processing method, comprising:

[0005] Obtain ultrasound images of the sagittal section of the fetal sacrococcygeal spine;

[0006] Input the ultrasound image into the pre-trained deep learning neural network model to obtain the section evaluation result of the ultrasound image;

[0007] If the section evaluation result meets the preset requirements, the conus medullaris terminal end, the sacrum and the lumbar vertebrae in the ultrasound image identified by the deep learning neural network model are marked in the ultrasound image to obtain a marked ultrasound image.

[0008] Preferably, it also includes:

[0009] Determine the number of vertebrae between the end of the sacrococcygeal spine and the end of the conus medullaris in the ultrasound image;

[0010] If the number of vertebrae is less than the preset threshold, an alarm message is generated.

[0011] Preferably, obtaining an ultrasound image of the sagittal section of the fetal sacral and coccygeal spine includes:

[0012] Acquire ultrasound images during fetal ultrasound testing and capture ultrasound images from the ultrasound images.

[0013] Preferably, based on the spinal conus medullaris terminal, the sacrum and the lumbar vertebrae in the ultrasound image identified by the deep learning neural network model, the spinal conus medullaris terminal, the sacrum and the lumbar vertebrae are marked in the ultrasound image, and before obtaining the marked ultrasound image, the method further includes:

[0014] Compare the section evaluation results with the previous section evaluation results;

[0015] If the section evaluation result is better than the previous section evaluation result, the step of identifying the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image based on the deep learning neural network model, marking the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image, and obtaining a marked ultrasound image is performed.

[0016] Preferably, after obtaining the annotated ultrasound image, the method further includes:

[0017] If a modification instruction is received, the annotated positions of the conus medullaris terminal, the sacral vertebrae and the lumbar vertebrae in the annotated ultrasound image are modified according to the modification instruction.

[0018] Preferably, the training process of the deep learning neural network model includes:

[0019] Obtaining a training set, and obtaining a target sample from the training set; wherein the target sample is annotated with a section category and a section score, and the target sample whose section category is annotated as a standard section also annotates the contours of the conus medullaris terminal, the sacral vertebrae, and the lumbar vertebrae;

[0020] The target sample is input into the initial model for training to obtain the pre-trained deep learning neural network model.

[0021] On the other hand, the present application also provides an ultrasonic image processing device, comprising:

[0022] An acquisition module, used for acquiring an ultrasound image of a sagittal section of the fetal sacral and coccygeal spine;

[0023] An evaluation module is used to input the ultrasound image into a pre-trained deep learning neural network model to obtain a section evaluation result of the ultrasound image;

[0024] The identification and annotation module is used to annotate the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image based on the deep learning neural network model to obtain an annotated ultrasound image if the section evaluation result meets the preset requirements.

[0025] Preferably, it also includes:

[0026] The vertebral body number counting module is used to determine the number of vertebral bodies between the end of the sacral and coccygeal spine and the end of the conus medullaris in the ultrasound image; if the number of vertebral bodies is less than a preset threshold, an alarm prompt message is generated.

[0027] Preferably, the acquisition module is specifically used for:

[0028] Acquire ultrasound images during fetal ultrasound testing and capture ultrasound images from the ultrasound images.

[0029] Preferably, it also includes:

[0030] A comparison module is used to compare the section evaluation result with the previous section evaluation result; if the section evaluation result is better than the previous section evaluation result, the step of performing the identification and annotation module to identify the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image based on the deep learning neural network model, annotate the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image, and obtain an annotated ultrasound image.

[0031] Preferably, it also includes:

[0032] The correction module is used for modifying the annotated positions of the conus medullaris terminal, the sacral vertebrae and the lumbar vertebrae in the annotated ultrasound image according to the modification instruction if a modification instruction is received.

[0033] Preferably, it also includes:

[0034] Model training module for training deep learning neural network models. The model training module includes:

[0035] An acquisition unit, used to acquire a training set and acquire a target sample from the training set; wherein the target sample is annotated with a section category and a section score, and the target sample whose section category is annotated as a standard section also annotates the contours of the conus medullaris terminal, sacral vertebrae, and lumbar vertebrae;

[0036] The training unit is used to input the target sample into the initial model for training to obtain the pre-trained deep learning neural network model.

[0037] On the other hand, the present application also provides an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned ultrasound image processing method.

[0038] Specifically, the electronic device may be an ultrasonic diagnostic device or an ultrasonic workstation.

[0039] On the other hand, the present application also provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the aforementioned ultrasound image processing method is implemented.

[0040] After obtaining the ultrasound image of the sagittal section of the fetal sacral and coccygeal spine, the present application can automatically evaluate the ultrasound image using a pre-trained deep learning neural network model to obtain the section evaluation result of the ultrasound image; if the section evaluation result meets the preset requirements, it indicates that the sagittal section of the fetal sacral and coccygeal spine in the ultrasound image is clear enough, and the spinal conus medullaris can be clearly seen with a high probability, so based on the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae in the ultrasound image identified by the deep learning neural network model, the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae are marked in the ultrasound image to obtain a marked ultrasound image. It can be seen that the scheme can automatically identify and mark the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae in the ultrasound image, so the obtained marked ultrasound image can help doctors assist in determining the position of the spinal conus medullaris terminal, improving the accuracy and efficiency of the examination.

[0041] Correspondingly, an ultrasonic image processing device, equipment and storage medium provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 A flow chart of an ultrasonic image processing method provided in this application;

[0044] Figure 2 A schematic diagram of an ultrasound image with annotations provided in the present application;

[0045] Figure 3 A flow chart of another ultrasound image processing method provided in this application;

[0046] Figure 4 A flow chart of a deep learning neural network model training method provided in this application;

[0047] Figure 5 A non-labeled cross-sectional schematic diagram provided for this application;

[0048] Figure 6 A schematic diagram of an ultrasonic image processing device provided in this application;

[0049] Figure 7 A server structure diagram provided for this application;

[0050] Figure 8 A terminal structure diagram provided for this application;

[0051] Fig. 9 A schematic diagram of the process of processing ultrasound images using a deep learning neural network model provided in this application. DETAILED DESCRIPTION

[0052] The traditional method of locating and identifying the end of the conus medullaris is that the doctor manually cuts the sagittal section of the sacral and coccygeal spine in the ultrasound image and visually inspects the position of the conus medullaris on this section. However, since there are many vertebrae on this section (usually about a dozen vertebrae) and they are densely packed, the visual inspection method is time-consuming and prone to errors.

[0053] In view of the above-mentioned problems, the present application proposes an ultrasound image processing solution, which can automatically identify and mark the distal end of the conus medullaris, sacral vertebrae and lumbar vertebrae in ultrasound images. Therefore, the obtained annotated ultrasound image can help doctors assist in determining the position of the distal end of the conus medullaris, thereby improving the accuracy and efficiency of the examination.

[0054] See also Figure 1 , Figure 1 The following is a flow chart of an ultrasonic image processing method provided in an embodiment of the present application. Figure 1 As shown, the ultrasound image processing method may include the following steps:

[0055] S101. Obtain an ultrasound image of the sagittal section of the fetal sacral and coccygeal spine.

[0056] The ultrasound image in this embodiment can be obtained by intercepting the ultrasound image frame by frame. Therefore, in a specific embodiment, obtaining the ultrasound image of the sagittal section of the fetal sacral and coccygeal spine includes: obtaining the ultrasound image during the fetal ultrasound detection process, and intercepting the ultrasound image from the ultrasound image.

[0057] S102: Input the ultrasound image into a pre-trained deep learning neural network model to obtain a section evaluation result of the ultrasound image.

[0058] In this embodiment, the deep learning neural network model can be: Mask R-CNN, Mask Scoring R-CNN, PA Net, etc. These models are deep learning neural network models based on instance segmentation. The specific model structure is not limited in this embodiment, that is, the convolution layer, pooling layer and other structures in the model can be increased or decreased according to actual conditions.

[0059] Among them, the section evaluation results are used to evaluate whether the sagittal section of the fetal sacrococcygeal spine in the ultrasound image is clear enough and whether the anatomical structure is complete. Only when the sagittal section of the fetal sacrococcygeal spine is clear enough and the anatomical structure is complete enough, can the spinal components such as the vertebral canal, conus medullaris, sacral vertebrae and lumbar vertebrae be seen. The section evaluation results can be expressed as a numerical value. The larger the value, the clearer and more standard the section. The deep learning neural network model can be used to evaluate the section of ultrasound images with the help of the binary classification idea. That is, the deep learning neural network model can classify the input ultrasound image into standard sections and non-standard sections, and give corresponding evaluation values.

[0060] In a specific embodiment, the specific process of evaluating the section evaluation result of the ultrasound image by the deep learning neural network model includes: the deep learning neural network model extracts image features in the ultrasound image, determines the region of interest (such as Figure 2 The area shown in the figure can be used to determine whether the section is standard based on the region of interest.

[0061] S103, determining whether the section evaluation result meets the preset requirements; if so, executing S104; if not, discarding the ultrasound image.

[0062] Among them, if the aspect evaluation result is expressed in numerical values, the preset requirement can be a specific threshold value. If the numerical value representing the aspect evaluation result is greater than the threshold value, the aspect evaluation result is considered to meet the preset requirement; otherwise, the aspect evaluation result is considered to not meet the preset requirement. If the aspect evaluation result ranges from 1 to 100, the preset requirement can be 90. If the aspect evaluation result ranges from 1 to 10, the preset requirement can be 9. It can be seen that the aspect evaluation result can be expressed in the form of aspect score and the like.

[0063] S104. Based on the deep learning neural network model, the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image are identified, and the distal end of the conus medullaris, the sacrum and the lumbar vertebrae are marked in the ultrasound image to obtain a marked ultrasound image.

[0064] Annotated ultrasound images can be found at Figure 2 .based on Figure 2 As shown, the relative position of the conus medullaris terminal and the sacral and lumbar vertebrae can be clearly seen. Figure 2 The conus medullaris end shown is located between L3 and L2, which can help doctors determine whether the conus medullaris is normal.

[0065] It should be noted that L1-L5 represent five lumbar vertebrae, i.e., five vertebrae located in the waist; S1-S5 represent five sacral vertebrae, i.e., five vertebrae located in the sacrum. The human spine is composed of multiple vertebrae, and one vertebrae is composed of two vertebral arches and one vertebral body, so one vertebral body corresponds to one vertebrae.

[0066] In a specific embodiment, the specific process of deep learning neural network model recognition of ultrasound images includes: the deep learning neural network model extracts image features in the target sample, determines the region of interest based on the image features; determines the spinal canal contour, spinal conus contour, and contours of each vertebra in the region of interest; generates a first image mask including the spinal canal contour, a second image mask including the spinal conus contour, and a third image mask including the contours of each vertebra; compares the first image mask, the second image mask, and the third image mask to identify the spinal conus terminal, sacral vertebrae, and lumbar vertebrae in the target sample. The image mask is a binary image that records the coordinate position of the contour.

[0067] It can be seen that after acquiring the ultrasound image of the sagittal section of the fetal sacral and coccygeal spine, this embodiment can automatically evaluate the ultrasound image using the pre-trained deep learning neural network model to obtain the section evaluation result of the ultrasound image; if the section evaluation result meets the preset requirements, it indicates that the sagittal section of the fetal sacral and coccygeal spine in the ultrasound image is clear enough, the anatomical structure is complete enough, and the spinal conus medullaris can be clearly seen with a high probability. Therefore, based on the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae in the ultrasound image identified by the deep learning neural network model, the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae are marked in the ultrasound image to obtain a marked ultrasound image. It can be seen that this scheme can automatically identify and mark the spinal conus medullaris terminal, sacral vertebrae and lumbar vertebrae in the ultrasound image, so the obtained marked ultrasound image can help doctors assist in determining the position of the spinal conus medullaris terminal, thereby improving the accuracy and efficiency of the examination.

[0068] See also Figure 3 , Figure 3 Another ultrasonic image processing method flow chart provided in the embodiment of the present application. Figure 3 As shown, the ultrasound image processing method is used in the process of fetal ultrasound detection, and may specifically include the following steps:

[0069] S301, obtaining an ultrasound image of the sagittal section of the fetal sacral and coccygeal spine.

[0070] S302: Input the ultrasound image into a pre-trained deep learning neural network model to obtain a section evaluation result of the ultrasound image.

[0071] S303, determining whether the section evaluation result meets the preset requirements; if so, executing S304; if not, discarding the ultrasound image.

[0072] S304, determining whether the section evaluation result is better than the previous section evaluation result; if so, executing S305; if not, discarding the ultrasound image.

[0073] The “previous section evaluation result” refers to the previous section evaluation result that meets the preset requirements, that is, the section evaluation result corresponding to the ultrasound image that has not been discarded after being processed in steps S303 and S304.

[0074] S305. Based on the deep learning neural network model, the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image are identified, and the distal end of the conus medullaris, the sacrum and the lumbar vertebrae are annotated in the ultrasound image to obtain an annotated ultrasound image.

[0075] The method provided in this embodiment is used in the process of fetal ultrasound detection, by continuously acquiring ultrasound images of the sagittal section of the fetal sacral and coccygeal spine, to obtain relatively clear and accurate ultrasound images with annotations, so as to help doctors to assist in determining the position of the end of the conus medullaris. That is, the ultrasound image with annotations obtained by the ultrasound device in the final scan is the clearest and most accurate in the entire detection process.

[0076] It can be seen that in order to obtain an ultrasound image with the best annotation effect, an ultrasound image with the best section evaluation result can be selected from multiple ultrasound images of a section of ultrasound image based on the section evaluation result. For example: 10 ultrasound images are obtained from the ultrasound image, and the 10 ultrasound images are first evaluated frame by frame using a deep learning neural network model. If the section evaluation result of the current ultrasound image evaluated meets the preset requirements, indicating that it is a standard section image (that is, the corresponding section score is greater than the threshold), the section evaluation result of the current ultrasound image is compared with the previous section evaluation result that meets the preset requirements. If the section evaluation result of the current ultrasound image is better than the previous section evaluation result that meets the preset requirements, S305 is executed to annotate the current ultrasound image; otherwise, the current ultrasound image is discarded. This cycle is repeated until the best ultrasound image is selected from the 10 ultrasound images. At this time, the annotation of ultrasound images that do not meet the requirements can be reduced, thereby saving computing resources.

[0077] In addition, the section evaluation results can be compared for the ultrasound images (i.e., ultrasound videos) of the same part obtained from the two previous scans. For example: the model selects the best standard section image (recorded as standard section image 1) from the first section of ultrasound images, and also selects the best standard section image (recorded as standard section image 2) from the second section of ultrasound images. At this time, the section evaluation results (i.e., section scores) of standard section image 1 and standard section image 2 can be compared. If the section score of standard section image 2 is greater than the section score of standard section image 1, standard section image 2 replaces standard section image 1; if less than, standard section image 1 is retained as the best standard section image for the current scanned part. Based on this, the best standard section can be selected from two or more sections of ultrasound images for annotation.

[0078] It can be seen that this embodiment can automatically identify and mark the distal end of the conus medullaris, sacral vertebrae and lumbar vertebrae in the ultrasound image during the ultrasound detection process, and can select the clearest and most accurate labeled ultrasound image output to help doctors assist in determining the position of the distal end of the conus medullaris, thereby improving the accuracy and efficiency of the examination.

[0079] Based on the above embodiment, it should be noted that in the late stage of fetal development, the number of vertebrae between the end of the sacrococcygeal spine (i.e., the tail of the spine) and the end of the conus medullaris will not be less than 6. Therefore, based on the annotated ultrasound image, the number of vertebrae between the end of the sacrococcygeal spine and the end of the conus medullaris in the ultrasound image can be determined; if the number of vertebrae is less than a preset threshold (such as: 6), an alarm prompt message is generated.

[0080] Among them, in order to accurately identify each vertebra and the end of the conus medullaris in the ultrasound image, the deep learning neural network model can use other means to correct the labeled position. For example, the distance between two adjacent vertebrae can be calculated. If the distance exceeds the normal value, it can be considered that one or more vertebrae may be missed between the two vertebrae. Alternatively, the contour of the conus medullaris can be identified and corrected with the help of the dark area in the spinal canal in the ultrasound image. The conus medullaris is in the spinal canal, and the conus medullaris is a dark area in the ultrasound image.

[0081] Based on the above embodiment, it should be noted that if there are obvious errors in the deep learning neural network model's annotation of the ultrasound image, the ultrasound doctor can also manually annotate or modify it. Therefore, after obtaining the annotated ultrasound image, it also includes: if a modification instruction is received, the annotation position of the spinal cord conus terminal, sacral vertebrae and lumbar vertebrae in the annotated ultrasound image is modified according to the modification instruction.

[0082] Among them, the annotations for the end of the conus medullaris, sacrum and lumbar vertebrae can be as follows: Figure 2 As shown, it can also be text, sound, etc.

[0083] See also Figure 4 , Figure 4 A flow chart of a deep learning neural network model training method provided in an embodiment of the present application. Figure 4 As shown, the method may specifically include the following steps:

[0084] S401, obtaining a training set.

[0085] The training set includes multiple ultrasound images of sagittal sections of the fetal sacral and coccygeal spine. Some are standard section training samples, which are labeled with information such as the end of the conus medullaris, the contours of the sacral and lumbar vertebrae, the section category, and the section score; some are non-standard section training samples, such as image samples that only show the fuzzy structure of the spine (such as Figure 5), which can be labeled with: section category, rectangular box covering spinal structure, and section score. These two types of samples are used by the model to learn the ability to evaluate the section of ultrasound images, that is, the ability to classify ultrasound images and locate target tissues.

[0086] Among them, the aspect categories can be divided into standard aspects and non-standard aspects. For example: Figure 2 The "Standard Section of Fetal Spine Sagittal Section" is shown. Figure 5 It illustrates the "non-standard sagittal section of the fetal spine".

[0087] S402, obtaining target samples from the training set; the target samples include the above-mentioned standard section training samples and non-standard section training samples.

[0088] In a specific implementation, before obtaining the target sample from the training set, it also includes: performing normalization processing on each target sample in the training set. The normalization processing includes: centering, standardization, normalization, etc., to standardize the style of the sample image, remove unnecessary interference information, such as noise information, and reduce the impact of image style differences, so as to normalize the model input and improve the model accuracy.

[0089] In a specific implementation, after normalizing each target sample in the training set, the method further includes: translating, rotating or scaling any target sample after normalization to amplify the number of samples and increase the number of samples.

[0090] S403: Input the target sample into the initial model for training to obtain a pre-trained neural network model.

[0091] In a specific implementation, the target sample is input into the initial model, and the section classification result and the region of interest segmentation result of the target sample can be obtained. The specific process includes: using the initial model to extract the image features in the target sample, determining the region of interest based on the image features, and outputting the section classification result and the region of interest segmentation result based on the region of interest. The section classification result includes the classification categories of standard sections and non-standard sections, and the target region of interest segmentation result is the segmentation result of the region of interest when the section classification result is the standard section.

[0092] During the training process, the slice recognition loss corresponding to the above slice classification results and the positioning loss corresponding to the above segmentation results are counted; among which, the slice recognition loss includes the calculation of bounding box loss and classification loss, and the positioning loss includes segmentation loss and key point distance loss. The relevant losses are jointly weighted to obtain a comprehensive loss, and gradient backpropagation is performed based on the comprehensive loss. If the comprehensive loss does not meet the model convergence conditions, the model parameters of the initial model are updated based on the comprehensive loss, and new target samples are obtained from the training set to iteratively train the updated initial model until the comprehensive loss meets the model convergence conditions.

[0093] As the model training progresses, the comprehensive loss of the initial model gradually decreases until convergence, and the model training stops. At this time, the optimal model parameters are retained, and the corresponding model is the deep learning neural network model that has been pre-trained as mentioned above.

[0094] It can be seen that according to the scheme provided in this embodiment, a deep learning neural network model for identifying and annotating ultrasound images can be trained, and the trained deep learning neural network model can be deployed in ultrasound equipment for use in fetal ultrasound testing. Of course, the model can also identify and annotate historical images stored in ultrasound equipment.

[0095] See also Figure 6 , Figure 6 A schematic diagram of an ultrasonic image processing device provided in an embodiment of the present application includes:

[0096] An acquisition module 601 is used to acquire an ultrasound image of a sagittal section of the fetal sacral and coccygeal spine;

[0097] An evaluation module 602 is used to input the ultrasound image into a pre-trained deep learning neural network model to obtain a section evaluation result of the ultrasound image;

[0098] The identification and annotation module 603 is used to annotate the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image based on the deep learning neural network model if the section evaluation result meets the preset requirements, so as to obtain an annotated ultrasound image.

[0099] In a specific embodiment, it also includes:

[0100] The vertebral body number counting module is used to determine the number of vertebral bodies between the end of the sacral and coccygeal spine and the end of the conus medullaris in the ultrasound image; if the number of vertebral bodies is less than a preset threshold, an alarm prompt message is generated.

[0101] In a specific implementation, the acquisition module is specifically used to:

[0102] Acquire ultrasound images during fetal ultrasound testing and capture ultrasound images from the ultrasound images.

[0103] In a specific embodiment, it also includes:

[0104] A comparison module is used to compare the section evaluation result with the previous section evaluation result; if the section evaluation result is better than the previous section evaluation result, the step of annotating the spinal conus medullaris, sacral vertebrae and lumbar vertebrae in the ultrasound image identified based on the deep learning neural network model, and obtaining an ultrasound image with annotations is performed.

[0105] In a specific embodiment, it also includes:

[0106] The correction module is used for modifying the annotated positions of the conus medullaris terminal, the sacral vertebrae and the lumbar vertebrae in the annotated ultrasound image according to the modification instruction if a modification instruction is received.

[0107] In a specific implementation, a model training module is also included, which is used to train a deep learning neural network model. The model training module includes:

[0108] An acquisition unit, used to acquire a training set and acquire a target sample from the training set; wherein the target sample is annotated with a section category and a section score, and the target sample whose section category is annotated as a standard section also annotates the contours of the conus medullaris terminal, sacral vertebrae, and lumbar vertebrae;

[0109] The training unit is used to input the target sample into the initial model for training to obtain the pre-trained deep learning neural network model.

[0110] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0111] It can be seen that this embodiment provides an ultrasound image processing device, which can automatically identify and mark the distal end of the conus medullaris, sacral vertebrae and lumbar vertebrae in ultrasound images. Therefore, the obtained labeled ultrasound image can help doctors assist in determining the position of the distal end of the conus medullaris, thereby improving the accuracy and efficiency of the examination.

[0112] Furthermore, the present application also provides an electronic device. The electronic device can be Figure 7 The server 50 shown may also be Figure 8 The terminal 60 shown is, for example, an ultrasonic diagnostic device. Figure 7 and Figure 8 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0113] Figure 7 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 is used to store a computer program, which is loaded and executed by the processor 51 to implement the relevant steps in the ultrasonic image processing disclosed in any of the aforementioned embodiments.

[0114] In this embodiment, the power supply 53 is used to provide working voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 55 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0115] In addition, the memory 52, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 521, a computer program 522 and data 523, etc. The storage method can be temporary storage or permanent storage.

[0116] The operating system 521 is used to manage and control the hardware devices and computer programs 522 on the server 50, so as to realize the operation and processing of the data 523 in the memory 52 by the processor 51, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the ultrasonic image processing method disclosed in any of the aforementioned embodiments, the computer program 522 can further include a computer program that can be used to complete other specific tasks. In addition to data such as ultrasonic images, the data 523 can also include data such as parameters of the model.

[0117] Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 60 may specifically include but is not limited to ultrasonic diagnostic equipment, ultrasonic workstations, smart phones, tablet computers, laptop computers or desktop computers, etc.

[0118] Generally, the terminal 60 in this embodiment includes: a processor 61 and a memory 62 .

[0119] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0120] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 62 is at least used to store the following computer program 621, wherein the computer program is loaded and executed by the processor 61, and the relevant steps in the ultrasound image processing method performed by the terminal side disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 62 may also include an operating system 622 and data 623, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 622 may include Windows, Unix, Linux, etc. Data 623 may include but is not limited to ultrasound images.

[0121] In some embodiments, the terminal 60 may further include a display screen 63 , an input / output interface 64 , a communication interface 65 , a sensor 66 , a power supply 67 , and a communication bus 68 .

[0122] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 60, and the terminal 60 may include more or less components than those shown in the figure.

[0123] Furthermore, the present application also discloses a storage medium, in which a computer executable instruction is stored. When the computer executable instruction is loaded and executed by a processor, the ultrasonic image processing method disclosed in any of the above embodiments is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0124] The following is a detailed introduction to the training method of the deep learning neural network model provided in this application and the method of using the model.

[0125] 1. The training process of deep learning neural network model.

[0126] (1) Sample generation.

[0127] In order to obtain a model with a high accuracy, a large number of samples are needed. A professional doctor will perform ultrasound images of a large number of fetal sacrococcygeal spine sagittal sections according to the Figure 2 or Figure 5 Mark on the ultrasound image. If the section in the ultrasound image is standard, Figure 2 Mark; otherwise, follow Figure 5 The spine is marked (i.e., only the area where the spine is located is drawn with a rectangular frame). After the professional doctor completes the marking, the authoritative chief physician can review the marking results. In this way, a large number of samples can be obtained.

[0128] (2) Sample preprocessing.

[0129] Each sample obtained in (1) is cropped to remove unnecessary interference information. The sample is reduced to a specified size to normalize the size of the model input data. The sample is normalized to normalize the pixel value of the sample to the range of 0 to 1 to improve the convergence speed.

[0130] It should be noted that if the data input to the model is large, the gradient passed to the input layer during back propagation will become very large, causing the model to easily exceed the optimal point of the parameters.

[0131] The preprocessed samples are randomly divided into training set, validation set and test set in a ratio of 8:1:1. In addition, data augmentation can be performed on the training set and validation set (which can be achieved by operations such as translation, rotation, and scaling) to increase the number of samples and solve the small sample problem.

[0132] (3) Determine the model structure and use the model to process samples to train the model.

[0133] Establish a deep learning neural network model based on instance segmentation. You can use models such as Mask R-CNN, Mask ScoringR-CNN, and PA Net.

[0134] See also Fig. 9 , Fig. 9 The example shows a process of processing ultrasound images with a deep learning neural network model. The sample is input to the model. First, the feature extraction layer in the model extracts features from the ultrasound image (including deep features and edge contour features), and then the deep features and edge contour features are fused. Then, the region of interest is determined based on the fused features. Based on the region of interest, the current ultrasound image can be classified to determine the contour of the conus medullaris, the vertebral canal, and the contours of each vertebra, and the image mask corresponding to each contour is output separately. Multiple samples (for example, 20) can be input to the model at one time.

[0135] In the above training process, the model first classifies the sample to determine the section score in the sample. The threshold is used to determine whether the section is standard. If the section score is greater than the threshold, the section is a standard section. If the section score is lower than the threshold, the section is a non-standard section. If the section in the sample is determined to be a standard section, the subsequent recognition and labeling steps are continued, and the section recognition loss corresponding to the section classification result and the positioning loss corresponding to the segmentation result are calculated to update the model. If the section in the sample is determined to be a non-standard section, there is no need to perform the labeling step. Only the section recognition loss corresponding to the section classification result needs to be calculated to update the model parameters. The labeling step can label the sacrum, lumbar vertebrae and the lower edge of the conus medullaris, and give a position hint of the end of the conus medullaris relative to each vertebra. The hint is not limited to text, symbols or voice.

[0136] If multiple samples are processed this time, the average loss of the slice recognition loss corresponding to the slice classification results of these samples and / or the average loss of the positioning loss corresponding to the segmentation results are calculated, the obtained average losses are jointly weighted to determine a comprehensive loss, and then gradient backpropagation is performed to update the model parameters.

[0137] As the model training progresses, the comprehensive loss gradually decreases until convergence, and the model training stops. At this time, the current parameters of the model are retained to obtain the trained model.

[0138] 2. The process of using the deep learning neural network model.

[0139] Deploy the trained model to the ultrasound device. If the fetus is subjected to ultrasound examination, an ultrasound image of the sagittal section of the fetal sacral and coccygeal spine can be obtained from the ultrasound image, and the ultrasound image can be input into the trained model. First, the model gives a section score for the ultrasound image. If the section score is higher than the threshold, the ultrasound image is annotated; otherwise, the ultrasound image is discarded and a new ultrasound image is selected. Among them, a labeling switch can also be set. If the doctor turns on the labeling switch, labeling can be performed, thereby achieving one-click labeling. In addition, users can also be supported to manually modify the labeling position.

[0140] If multiple ultrasound images to be annotated are obtained during the detection process, the one with the highest section score is selected and displayed on the ultrasound device.

[0141] It can be seen that during the process of ultrasound examination of the fetus, the ultrasound equipment deployed with the trained model can automatically obtain the standard sagittal section of the fetal sacral and coccygeal spine, and automatically mark the sacral vertebrae, lumbar vertebrae and the lower edge of the conus medullaris. Based on this marking, the position of the lower edge of the conus medullaris and the spine can be accurately and quickly determined, which assists the doctor's judgment, improves the doctor's examination efficiency, and improves the accuracy of the examination.

[0142] It should be pointed out that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0143] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0144] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An ultrasonic image processing method, characterized in that: include: Obtain ultrasound images of the sagittal section of the fetal sacrococcygeal spine; Inputting the ultrasound image into a pre-trained deep learning neural network model to obtain a section evaluation result of the ultrasound image; If the section evaluation result meets the preset requirements, based on the spinal conus medullaris terminal, the sacrum and the lumbar vertebrae in the ultrasound image identified by the deep learning neural network model, the spinal conus medullaris terminal, the sacrum and the lumbar vertebrae are marked in the ultrasound image to obtain a marked ultrasound image; wherein, an ultrasound image with the best section evaluation result is selected from a plurality of ultrasound images of a segment of ultrasound image; Accordingly, the method further includes: marking the conus medullaris terminal, the sacrum and the lumbar vertebrae in the ultrasound image based on the deep learning neural network model, and obtaining the labeled ultrasound image before marking the conus medullaris terminal, the sacrum and the lumbar vertebrae in the ultrasound image. comparing the section evaluation result with a previous section evaluation result; If the section evaluation result is better than the previous section evaluation result, the step of identifying the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image based on the deep learning neural network model, marking the distal end of the conus medullaris, the sacrum and the lumbar vertebrae in the ultrasound image, and obtaining a labeled ultrasound image is performed; otherwise, the current ultrasound image is discarded; and this cycle is repeated until the best ultrasound image is selected.

2. The method according to claim 1, characterized in that Also includes: Determining the number of vertebrae between the end of the sacral and coccygeal spine and the end of the conus medullaris in the ultrasound image; If the number of vertebrae is less than a preset threshold, an alarm prompt message is generated.

3. The method according to claim 1, characterized in that The method of obtaining an ultrasound image of a sagittal section of the fetal sacral and coccygeal spine comprises: An ultrasonic image is acquired during a fetal ultrasonic detection process, and the ultrasonic image is captured from the ultrasonic image.

4. The method according to claim 1, characterized in that: After obtaining the annotated ultrasound image, the method further includes: If a modification instruction is received, the annotated positions of the conus medullaris terminal, the sacral vertebrae and the lumbar vertebrae in the annotated ultrasound image are modified according to the modification instruction.

5. The method according to any one of claims 1 to 4, characterized in that: The training process of the deep learning neural network model includes: Obtaining a training set, and obtaining a target sample from the training set; wherein the target sample is annotated with a section category and a section score, and the target sample whose section category is annotated as a standard section also annotates the contours of the conus medullaris terminal, the sacral vertebrae, and the lumbar vertebrae; The target sample is input into the initial model for training to obtain the pre-trained deep learning neural network model.

6. An ultrasonic image processing device, characterized in that: include: An acquisition module, used for acquiring an ultrasound image of a sagittal section of the fetal sacral and coccygeal spine; An evaluation module, used for inputting the ultrasound image into a pre-trained deep learning neural network model to obtain a section evaluation result of the ultrasound image; an identification and annotation module, configured to annotate the conus medullaris terminal, the sacrum and the lumbar vertebrae in the ultrasound image based on the deep learning neural network model identified therein if the section evaluation result meets the preset requirements, so as to obtain an ultrasound image with annotations; A comparison module is used to select an ultrasound image with the best section evaluation result from multiple ultrasound images of a section of ultrasound image; compare the section evaluation result with the previous section evaluation result; if the section evaluation result is better than the previous section evaluation result, execute the steps described in the recognition and annotation module to identify the end of the spinal cord conus, sacral vertebrae and lumbar vertebrae in the ultrasound image based on the deep learning neural network model, annotate the end of the spinal cord conus, the sacral vertebrae and the lumbar vertebrae in the ultrasound image, and obtain an annotated ultrasound image; otherwise, discard the current ultrasound image; and repeat this cycle until the best ultrasound image is selected.

7. The device according to claim 6, characterized in that Also includes: A vertebral body number counting module, used to determine the number of vertebral bodies between the end of the sacral and coccygeal spine and the end of the conus medullaris in the ultrasound image; If the number of vertebrae is less than a preset threshold, an alarm prompt message is generated.

8. The device according to claim 6, characterized in that The acquisition module is specifically used for: An ultrasonic image is acquired during a fetal ultrasonic detection process, and the ultrasonic image is captured from the ultrasonic image.

9. The device according to claim 6, characterized in that Also includes: The correction module is used for modifying the marked positions of the conus medullaris terminal, the sacral vertebrae and the lumbar vertebrae in the annotated ultrasound image according to the modification instruction if a modification instruction is received.

10. The device according to any one of claims 6 to 9, characterized in that Also includes: A model training module for training a deep learning neural network model, the model training module comprising: An acquisition unit, used to acquire a training set and acquire a target sample from the training set; wherein the target sample is annotated with a section category and a section score, and the target sample whose section category is annotated as a standard section also annotates the contours of the conus medullaris terminal, sacral vertebrae, and lumbar vertebrae; The training unit is used to input the target sample into the initial model for training to obtain the pre-trained deep learning neural network model.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 5.

12. The electronic device according to claim 11, characterized in that: The electronic equipment is an ultrasonic diagnostic device or an ultrasonic workstation.

13. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 5 is implemented.

Images