Bone segmentation and value measurement method based on fetus limb multi-section ultrasonic image

Abstract

The invention relates to the technical field of medical image analysis, in particular to a bone segmentation and value measurement method based on multi-section ultrasonic images of four limbs of a fetus, which comprises the following steps: acquiring and marking multi-section ultrasonic image data covering the whole development process of the fetus to construct a segmentation and measurement data set; introducing a semantic alignment jump connection module and a feature selection module into a preset skeleton segmentation neural network framework to obtain a skeleton segmentation neural network; according to the segmentation and measurement data set, optimizing the segmentation and measurement data set to obtain a skeleton structure segmentation model; receiving the segmentation result by using a preset bone length measurement module to extract a measurement endpoint coordinate; and constructing an end-to-end neural network model according to the skeleton structure segmentation model and the skeleton length value measurement module, and deploying the end-to-end neural network model in a target clinical ultrasound image analysis system. Therefore, the problems that in the prior art, data labeling is deficient, ultrasonic noise interference is serious, a segmentation measurement module lacks collaborative optimization, and measurement endpoint accuracy is insufficient are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image analysis, in particular to a method for fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement. BACKGROUND

[0002] The development of fetal long bones is one of the important bases for prenatal assessment of fetal development status and identification of lethal skeletal dysplasia (such as lethal bone dysplasia). Traditional prenatal ultrasound diagnosis mainly relies on manual image analysis by ultrasound doctors to judge the development shape and length of long bones, and then make a diagnosis combined with clinical experience. This method is not only time-consuming and labor-intensive, but also greatly influenced by the personal experience of the doctor, making it difficult to ensure the consistency of the diagnostic criteria and the accuracy of the measurement results.

[0003] In recent years, deep learning has been widely applied in the field of medical image analysis, especially in tasks such as organ segmentation and tumor identification. However, there are still many challenges in the segmentation and measurement of fetal ultrasound images of long bones, making it difficult for existing artificial intelligence methods to be directly applied to clinical use scenarios. Most existing public fetal ultrasound image datasets only provide target detection or rough bounding box annotations, lacking precise pixel-level segmentation annotations for fetal long bones (especially complex structures such as tibiofibular bones and radioulnar bones), which cannot support high-quality semantic segmentation and length measurement model training. In addition, fetal long bone images have significant differences due to factors such as fetal position, bone posture, and ultrasound device settings, resulting in significant changes in bone position, direction, and shape in the image, making segmentation tasks more difficult. Fetal ultrasound images have a low signal-to-noise ratio and are easily affected by artifacts and speckle noise, making it difficult for general image segmentation models such as U-Net, UNet++, or TransUNet based on Transformer to stably extract key edge information of the bone region. Furthermore, although the classic U-Net structure introduces a skip connection to fuse shallow and deep features, it often directly concatenates features with inconsistent semantics, introducing redundant information and reducing segmentation accuracy. In the ultrasound image measurement stage, traditional methods rely on edge fitting or manual point annotation to determine the ends of the bone, which can result in significant endpoint errors due to different habits of annotating doctors, affecting the consistency of the measurement results. Point regression methods based on deep neural networks also lack the ability to learn from clinical annotation patterns, making it difficult to achieve acceptable measurement accuracy in practical applications.

[0004] Therefore, there is an urgent need for a bone segmentation and measurement method that can overcome the problems of insufficient data annotation, severe ultrasound noise interference, lack of collaborative optimization of segmentation and measurement modules, and insufficient accuracy of measurement endpoints. SUMMARY

[0005] The application provides a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method to solve the problems of insufficient data annotation, serious ultrasound noise interference, lack of collaborative optimization of segmentation and measurement modules, and insufficient accuracy of measurement endpoints in the prior art.

[0006] The first aspect embodiment of the application provides a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method, comprising the following steps: collecting multi-slice ultrasound image data covering the entire development process of a fetus, and annotating the multi-slice ultrasound image data to construct a segmentation and measurement data set; introducing a semantic alignment skip connection module and a feature selection module into a bone segmentation neural network framework built in advance based on U-Net to obtain a bone segmentation neural network; optimizing the bone segmentation neural network framework according to the segmentation and measurement data set to obtain a bone structure segmentation model; receiving the segmentation result of the bone structure segmentation model by using a pre-constructed bone length measurement module to extract measurement endpoint coordinates; constructing an end-to-end neural network architecture according to the bone structure segmentation model and the bone length measurement module, and jointly optimizing and training the end-to-end neural network architecture by using a pre-constructed hybrid objective function to obtain an end-to-end neural network model; deploying the end-to-end neural network model in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output fetal long bone segmentation regions and bone length measurement values.

[0007] Optionally, the collecting multi-slice ultrasound image data covering the entire development process of a fetus, and annotating the multi-slice ultrasound image data to construct a segmentation and measurement training data set comprises:

[0008] The multi-slice ultrasound image data covering the entire development process of a fetus is collected, wherein the multi-slice ultrasound image data covering the entire development process of a fetus includes scanning section images corresponding to femurs, humerus, tibia and fibula, and radius and ulna in the four limbs of a fetus; a pixel-level segmentation mask annotation is performed on the bone region in each scanning section image, and measurement endpoints at both ends of the bone are annotated to construct the segmentation and measurement training data set.

[0009] Optionally, the introducing a semantic alignment skip connection module and a feature selection module into a bone segmentation neural network framework built in advance based on U-Net to obtain a bone segmentation neural network comprises:

[0010] The bone segmentation neural network framework is built based on a classic U-Net structure; the semantic alignment skip connection module is embedded in each level of skip connection in the bone segmentation neural network framework to reconstruct the projected features through cross-channel attention and space-channel attention mechanisms; and the feature selection module is added to the bottleneck layer in the bone segmentation neural network framework to filter out noise and amplify bone features by using a dynamic threshold.

[0011] Optionally, the skeleton segmentation neural network framework is optimized according to the segmentation and measurement dataset to obtain a skeleton structure segmentation model, including:

[0012] The classification loss and the region matching loss of the skeleton segmentation neural network are set as a joint optimization target; the segmentation and measurement dataset is preprocessed to obtain a preprocessed segmentation and measurement dataset; and the preprocessed segmentation and measurement dataset is used to optimize the skeleton segmentation neural network framework based on a phased training manner until the joint optimization target converges, so as to obtain the skeleton structure segmentation model.

[0013] Optionally, the segmentation result of the skeleton structure segmentation model is received by using a pre-constructed skeleton length measurement module to extract measurement endpoint coordinates, including:

[0014] The segmentation result of the skeleton structure segmentation model is received by using a pre-constructed skeleton length measurement module to extract initial skeleton endpoints; the segmentation result is processed by using a pre-constructed point regression mapping module through a depth residual block, multi-scale feature down-sampling, global average pooling and a full connection layer in sequence to predict endpoint offset; and the measurement endpoint coordinates are generated according to the initial skeleton endpoints and the endpoint offset.

[0015] Optionally, the end-to-end neural network model is deployed in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output fetal long bone segmentation regions and skeleton length measurement values, including:

[0016] The end-to-end neural network model is converted into a lightweight inference engine format and deployed in edge computing or an ultrasound workstation device of the target clinical ultrasound image analysis system, so as to receive original ultrasound frames of the real-time fetal ultrasound images through a standardized API interface and output the fetal long bone segmentation regions and the skeleton length measurement values.

[0017] In a second aspect, an embodiment of the present application provides a device for skeleton segmentation and measurement based on multi-slice ultrasound images of a fetus' limbs, comprising: a labeling module configured to collect multi-slice ultrasound image data covering the whole development process of a fetus, and label the multi-slice ultrasound image data to construct a segmentation and measurement dataset; a network construction module configured to introduce a semantic alignment skip connection module and a feature selection module into a skeleton segmentation neural network framework pre-built with a U-Net to obtain a skeleton segmentation neural network; a network optimization module configured to optimize the skeleton segmentation neural network framework according to the segmentation and measurement dataset to obtain a skeleton structure segmentation model; an endpoint measurement module configured to receive segmentation results of the skeleton structure segmentation model by using a pre-built skeleton length measurement module to extract measurement endpoint coordinates; a joint optimization module configured to construct an end-to-end neural network architecture according to the skeleton structure segmentation model and the skeleton length measurement module, and perform joint optimization training on the end-to-end neural network architecture by using a pre-built hybrid objective function to obtain an end-to-end neural network model; and a deployment module configured to deploy the end-to-end neural network model in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output a fetal long bone segmentation region and a skeleton length measurement value.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for skeleton segmentation and measurement based on multi-slice ultrasound images of a fetus' limbs as described in the above embodiment.

[0019] In a fourth aspect, an embodiment of the present application provides a computer program product, wherein the computer program / instructions are executed by a processor to implement the method for skeleton segmentation and measurement based on multi-slice ultrasound images of a fetus' limbs as described above.

[0020] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the method for skeleton segmentation and measurement based on multi-slice ultrasound images of a fetus' limbs as described above.

[0021] The method for fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement provided by the embodiment of the application, constructs a high-quality multi-slice fetal long bone ultrasound image dataset, and introduces a semantic alignment jump connection module and a positive sample adaptive sampling module on the basis of a classic U-Net architecture, to train a segmentation network with strong generalization ability, effectively alleviates the semantic difference between the encoder and the decoder through the improved network structure, to suppress the influence of image noise and improve the long bone segmentation precision; the point regression mapping strategy is adopted to learn the annotation offset of the clinical expert, to realize the iterative optimization of the initial bone endpoints and improve the measurement consistency and accuracy; through the weighted combination of cross-entropy loss, Dice loss and mean square error loss, the end-to-end joint training of segmentation and measurement tasks is completed, so that the model has high precision, light weight and real-time performance.

[0022] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and / or additional aspects and advantages of the application will become apparent and be readily understood by considering the following detailed description, including the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method provided by an embodiment of the application;

[0025] Figure 2 A flowchart of a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method provided by an embodiment of the application;

[0026] Figure 3 A flowchart of a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method provided by an embodiment of the application;

[0027] Figure 4 A block diagram of a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement device provided by an embodiment of the application;

[0028] Figure 5 A structural diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION

[0029] Embodiments of the application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the application, and cannot be understood as limiting the application.

[0030] Embodiments of the application are described below with reference to the accompanying drawings based on a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method.

[0031] Figure 1 A flowchart of a fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method provided by embodiments of the application.

[0032] As Figure 1 shown, the fetal four-limb multi-slice ultrasound image-based bone segmentation and measurement method includes the following steps:

[0033] In step S101, multi-slice ultrasound image data covering the entire fetal development process is collected, and the multi-slice ultrasound image data is labeled to construct a segmentation and measurement data set.

[0034] In some embodiments, multi-slice ultrasound image data covering the entire fetal development process is collected, and the multi-slice ultrasound image data is labeled to construct a segmentation and measurement training data set, including:

[0035] Collecting multi-slice ultrasound image data covering the entire fetal development process, wherein the multi-slice ultrasound image data covering the entire fetal development process includes scanning section images corresponding to femur, humerus, tibia and fibula, radius and ulna in the fetal limbs;

[0036] Performing pixel-level segmentation mask labeling on the bone region in each scanning section image, and labeling the measurement endpoints at both ends of the bone to construct a segmentation and measurement training data set.

[0037] In actual execution, multi-slice ultrasound image data covering the entire fetal development process is collected, focusing on standard scanning sections corresponding to femur, humerus, tibia and fibula, radius and ulna in the fetal limbs. Under the guidance of clinical experts, pixel-level segmentation mask labeling is performed on the bone region in each of the above images to form segmentation labels, and measurement endpoints at both ends of the bone are labeled to finally form a training data set corresponding one-to-one to segmentation and measurement.

[0038] In step S102, a semantic alignment skip connection module and a feature selection module are introduced into a bone segmentation neural network framework previously built with U-Net to obtain a bone segmentation neural network.

[0039] In some embodiments, a semantic alignment skip connection module and a feature selection module are introduced into a bone segmentation neural network framework previously built with U-Net to obtain a bone segmentation neural network, including:

[0040] Building a bone segmentation neural network framework based on a classic U-Net structure;

[0041] In each level of the skip connection of the skeleton segmentation neural network framework, a semantic alignment skip connection module is embedded to reconstruct the projected features through cross-channel attention and spatial-channel attention mechanisms.

[0042] In the bottleneck layer of the skeleton segmentation neural network framework, a feature selection module is added to filter out noise and amplify the skeleton features by using a dynamic threshold.

[0043] In actual execution, a skeleton segmentation neural network framework based on a U-Net encoder-decoder is built, a semantic alignment skip connection (SASC) module is embedded in each level of the skip connection, and the projected features are reconstructed through cross-channel attention (CCA) and spatial-channel attention (SCA) mechanisms to reduce the semantic difference between the encoder and the decoder and improve the semantic consistency and boundary clarity between the encoder and the decoder. Meanwhile, a positive sample adaptive sampling (PoSamp) module (i.e., a feature selection module) is added to the bottleneck layer to reduce noise and amplify the skeleton features of the high-dimensional information of the bottleneck layer, so as to retain key information, enhance the sensitivity of the model to the skeleton target, and improve the segmentation accuracy.

[0044] In step S103, the skeleton segmentation neural network framework is optimized according to the segmentation and measurement dataset to obtain a skeleton structure segmentation model.

[0045] In some embodiments, the skeleton segmentation neural network framework is optimized according to the segmentation and measurement dataset to obtain a skeleton structure segmentation model, including:

[0046] The classification loss and the region matching loss of the skeleton segmentation neural network are set as a joint optimization target;

[0047] The segmentation and measurement dataset is preprocessed to obtain a preprocessed segmentation and measurement dataset;

[0048] Based on the phased training method, the preprocessed segmentation and measurement dataset is used to optimize the skeleton segmentation neural network framework until the joint optimization target converges, and a skeleton structure segmentation model is obtained.

[0049] In actual execution, the original images in the segmentation and measurement dataset and their corresponding labels are preprocessed, wherein the preprocessing includes operations such as size standardization, channel alignment, normalization and multi-channel mask expansion, so as to represent the target region in the image by multi-channel coding, split the segmentation mask into multiple channels to represent different bone categories, ensure the channel alignment and category distinction of the network input, ensure that the network can identify different types of long bone structures and their boundary features, provide adaptive input for the subsequent training of the segmentation network, and generally divide the preprocessed dataset into a training set and a validation set at a ratio of 8:2. At the same time, image enhancement operations such as random rotation, horizontal flip, brightness disturbance, addition of Gaussian noise and the like are used to expand sample diversity, thereby improving the robustness of the model to actual complex image environments.

[0050] The weighted combination of the classification loss and the region matching Dice loss is set as the joint optimization target, based on the phased training method, the SGD optimizer is used, the initial learning rate is set to 2e-2, the batch size is 8, supplemented by Dropout and early stopping strategy, the training set is used to optimize the skeleton segmentation neural network framework until the joint optimization target converges, and a skeleton structure segmentation model is obtained.

[0051] In step S104, the segmentation result of the skeleton structure segmentation model is received by using a pre-constructed bone length measurement module to extract measurement endpoint coordinates.

[0052] In some embodiments, the segmentation result of the skeleton structure segmentation model is received by using a pre-constructed bone length measurement module to extract measurement endpoint coordinates, including:

[0053] The segmentation result of the skeleton structure segmentation model is received by using a pre-constructed bone length measurement module to extract initial bone endpoint;

[0054] The segmentation result is processed by using a pre-constructed point regression mapping module in sequence through a depth residual block, multi-scale feature downsampling, global average pooling and a fully connected layer to predict endpoint offset;

[0055] The measurement endpoint coordinates are generated according to the initial bone endpoint and the endpoint offset.

[0056] In actual execution, a skeleton length measurement module is constructed based on the skeleton structure segmentation model, the skeleton length measurement module receives the segmentation result of the skeleton structure segmentation model, automatically extracts a skeleton connected domain and initializes a skeleton endpoint Pini; a point regression mapping module based on ResNet50 is used to realize deep feature extraction in the form of residual unit stacking; spatial semantic information of the skeleton structure is obtained through multi-scale downsampling operations, the overall representation of the skeleton region is extracted through a global average pooling layer, and finally the endpoint offset of the key endpoint coordinates is output through a fully connected layer; the initialized skeleton endpoint is optimized using the endpoint offset, so as to obtain measurement endpoint coordinates closer to the labeling habits of clinicians.

[0057] In step S105, an end-to-end neural network architecture is constructed according to the skeleton structure segmentation model and the skeleton length measurement module, and a hybrid objective function is constructed to jointly optimize and train the end-to-end neural network architecture, so as to obtain an end-to-end neural network model.

[0058] In actual execution, the skeleton structure segmentation model and the skeleton length measurement module are jointly composed into an end-to-end neural network architecture; a hybrid objective function is constructed by fusing the weighted sum of the classification loss, the segmentation overlap Dice loss and the point regression error MSE loss, specifically, a cross-entropy loss function is used for image classification region supervision, a segmentation overlap Dice coefficient loss function is used for mask contour similarity evaluation, and a mean square error loss function is used for measurement endpoint coordinate regression supervision; based on the AdamW optimizer, an initial learning rate of 5e-4 and a batch size of 16, the hybrid objective function is used to jointly optimize and train the end-to-end neural network architecture, so as to realize collaborative optimization in the segmentation and measurement tasks, and thus the end-to-end neural network model reaches an optimized result in terms of segmentation accuracy and measurement accuracy.

[0059] In step S106, the end-to-end neural network model is deployed in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output fetal long bone segmentation regions and bone length measurements.

[0060] In some embodiments, the end-to-end neural network model is deployed in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output fetal long bone segmentation regions and bone length measurements, including:

[0061] The end-to-end neural network model is converted into a lightweight inference engine format and deployed in an edge computing or ultrasound workstation device of a target clinical ultrasound image analysis system, receives original ultrasound frames of real-time fetal ultrasound images through a standardized API interface, and outputs fetal long bone segmentation regions and bone length measurements.

[0062] In actual execution, the trained end-to-end neural network model is converted into a lightweight inference engine format, supporting mainstream model deployment frameworks such as ONNX and TorchScript, and is deployed in the edge computing or ultrasound workstation of the target clinical ultrasound image analysis system. The system receives real-time fetal ultrasound images through a standardized API interface, automatically outputs fetal long bone segmentation regions and bone length measurements, and meets the needs of real-time clinical applications, providing intelligent assistance for prenatal bone development evaluation and abnormality screening.

[0063] Further, the target clinical ultrasound image analysis system deploying the end-to-end neural network model has an image quality evaluation mechanism that can detect whether the image has abnormal conditions such as occlusion, blur, or artifacts, and issue a prompt when the quality is below a preset threshold. At the same time, the bone measurements of consecutive image frames are smoothed, and if the measurements fluctuate abnormally, a consistency verification module is automatically triggered for correction. The system also supports remote updates, model iteration upgrades, and one-key export of clinical measurement reports, improving the intelligence and clinical practicality of the method.

[0064] As shown in Figures 2-3 The following further describes the bone segmentation and measurement method based on fetal four-limb multi-slice ultrasound images according to an embodiment of the present application through a specific embodiment.

[0065] Step 1: According to the clinical ultrasound examination specification, standard multi-slice ultrasound images containing femur, humerus, tibia and fibula are collected, covering different gestational weeks. A number of ultrasound experts with more than ten years of experience perform pixel-level bone segmentation labeling on each image, and label the bone endpoints on the segmentation mask to form training samples corresponding to segmentation and measurement. For the same image, the expert needs to repeatedly check the mask and endpoint position for more than three times to ensure consistency; the labeling results are cross-checked, and samples with obvious deviations are removed, finally obtaining 1690 high-quality labeled images.

[0066] Step 2: The collected ultrasound raw images and their corresponding masks are uniformly normalized and scaled in size, adjusting all images to 512x512 pixels; the masks of the four types of bones are mapped to independent channels respectively to realize multi-channel input. The data enhancement process includes random rotation ±15°, translation ±10 pixels, horizontal flip, contrast adjustment and addition of Gaussian noise to enhance the robustness of the model to image acquisition angle and noise.

[0067] Step 3, a skeleton segmentation network is constructed based on the U-Net architecture, wherein the encoder and the decoder each consist of five convolutional blocks, each of which contains two 3x3 convolutions and one 2x2 max pooling or up-sampling. A semantic alignment skip connection (SASC) module is inserted in each level of the skip connection, which fuses and reconstructs the projected high-level and low-level features by cross-channel attention and spatial-channel attention mechanisms to reduce semantic differences; a positive sample adaptive sampling (PoSamp) module is added at the bottleneck layer to dynamically select high-response features and fuse them with the original features to further suppress ultrasound noise and highlight the bone edges.

[0068] Step 4, the preprocessed data is divided into a training set and a validation set at a ratio of 80:20, and a cross-entropy loss and a Dice loss are combined as the optimization objective; an SGD optimizer is used with an initial learning rate of 0.02, a momentum of 0.9, a batch size of 8, and 100 rounds of training; and early stopping monitoring is performed on the validation set, and the training is terminated when the validation loss does not decrease for 10 consecutive rounds. During the training process, for a small number of samples with low segmentation accuracy, an online hard example mining strategy is used for focused updating to further improve the segmentation capability of the model for difficult sections and low-contrast bones.

[0069] Step 5, the mask output by the segmentation network trained in step 4 is used to automatically extract the bone connected region and calculate the initial endpoint position Pini; a point regression mapping (PRM) module based on ResNet50 is constructed, which contains three residual bottleneck blocks, multi-scale down-sampling, global average pooling, and a fully connected layer in sequence, for predicting the endpoint offset and adding it to Pini to obtain the final measurement endpoint.

[0070] Step 6, the segmentation network and the measurement network are connected in series to form an end-to-end network; a hybrid loss function is constructed by combining the cross-entropy loss, the Dice loss, and the endpoint regression error (MSE), and an AdamW optimizer is used with an initial learning rate of 5e-4 and a batch size of 16 to fine-tune the entire network for 30 rounds to achieve collaborative optimization of the segmentation and measurement tasks and obtain the best overall performance.

[0071] Step 7, the trained end-to-end model is converted to ONNX or TorchScript format and deployed on an edge computing platform or an ultrasound workstation; real-time ultrasound frames are obtained through a standardized interface, and segmentation masks and measurement lengths of four types of long bones are output, with the measurement results displayed in millimeters and visualized overlays provided on the interface to meet the needs of real-time clinical evaluation.

[0072] In summary, the method for skeleton segmentation and measurement based on fetal four-limb multi-section ultrasound images according to the embodiments of the present application has the following advantages:

[0073] (1) The application can stably extract bone edge features in a complex noise environment by introducing a semantic alignment and adaptive sampling mechanism in the segmentation network, greatly reducing the misclassification and omission phenomenon, and significantly improving the Dice and IoU indicators of the segmentation model, ensuring high precision and high consistency of the segmentation results;

[0074] (2) Combined with the point regression mapping strategy, the expert annotation bias can be automatically learned to realize accurate positioning of the bone endpoints. Compared with the traditional edge fitting or manual annotation method, the measurement error is significantly reduced, the measurement result is more in line with the actual clinical needs, and the uncertainty caused by human intervention is reduced;

[0075] (3) The end-to-end lightweight design is adopted, and no additional post-processing is required in the training and inference process. The model has fast inference speed and low resource occupation, can be seamlessly integrated into the ultrasound instrument and edge computing platform, realizes real-time and automatic bone segmentation and measurement, greatly improves the clinical work efficiency, reduces the operation burden of medical staff, and is suitable for clinical real-time auxiliary diagnosis scene.

[0076] Secondly, the bone segmentation and measurement device based on fetal four-limb multi-section ultrasonic image according to the embodiment of the application is described with reference to the accompanying drawings.

[0077] Figure 4 The block schematic diagram of the bone segmentation and measurement device based on fetal four-limb multi-section ultrasonic image provided by the embodiment of the application is shown.

[0078] As shown in Figure 4 The bone segmentation and measurement device based on fetal four-limb multi-section ultrasonic image 40 includes a labeling module 401, a network construction module 402, a network optimization module 403, an endpoint measurement module 404, a joint optimization module 405, and a deployment module 406.

[0079] The marking module 401 is configured to collect multi-slice ultrasound image data covering the whole process of fetal development, and mark the multi-slice ultrasound image data to construct a segmentation and measurement data set. The network construction module 402 is configured to introduce a semantic alignment skip connection module and a feature selection module into a skeleton segmentation neural network framework built in advance based on a U-Net to obtain a skeleton segmentation neural network. The network optimization module 403 is configured to optimize the skeleton segmentation neural network framework based on the segmentation and measurement data set to obtain a skeleton structure segmentation model. The endpoint measurement module 404 is configured to receive a segmentation result of the skeleton structure segmentation model by using a pre-constructed skeleton length measurement module to extract measurement endpoint coordinates. The joint optimization module 405 is configured to construct an end-to-end neural network architecture based on the skeleton structure segmentation model and the skeleton length measurement module, and perform joint optimization training on the end-to-end neural network architecture by using a pre-constructed hybrid objective function to obtain an end-to-end neural network model. The deployment module 406 is configured to deploy the end-to-end neural network model in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output a fetal long bone segmentation region and a bone length measurement value.

[0080] In some embodiments, the marking module 401 comprises:

[0081] The collection unit is configured to collect multi-slice ultrasound image data covering the whole process of fetal development, wherein the multi-slice ultrasound image data covering the whole process of fetal development includes scanning section images corresponding to femurs, humerus, tibia and fibula, and radius and ulna in fetal limbs.

[0082] The marking unit is configured to perform pixel-level segmentation mask marking on a bone region in each scanning section image, and mark measurement endpoints at both ends of the bone to construct a segmentation and measurement training data set.

[0083] In some embodiments, the network construction module 402 comprises:

[0084] The building unit is configured to build a skeleton segmentation neural network framework based on a classic U-Net structure.

[0085] The embedding unit is configured to embed a semantic alignment skip connection module in each level of skip connection of the skeleton segmentation neural network framework to reconstruct the projected features by using a cross-channel attention and a space-channel attention mechanism.

[0086] The adding unit is configured to add a feature selection module to a bottleneck layer in the skeleton segmentation neural network framework to filter out noise and amplify bone features by using a dynamic threshold.

[0087] In some embodiments, the network optimization module 403 comprises:

[0088] The setting unit is configured to set a classification loss and a region matching loss of the skeleton segmentation neural network as a joint optimization target.

[0089] The preprocessing unit is configured to preprocess the segmentation and measurement dataset to obtain a preprocessed segmentation and measurement dataset.

[0090] The optimization unit is configured to optimize the skeleton segmentation neural network framework based on a staged training manner using the preprocessed segmentation and measurement dataset until the joint optimization target converges, to obtain a skeleton structure segmentation model.

[0091] In some embodiments, the endpoint measurement module 404 includes:

[0092] The receiving unit is configured to receive a segmentation result of the skeleton structure segmentation model using a pre-constructed bone length measurement module to extract initial bone endpoints.

[0093] The processing unit is configured to process the segmentation result using a pre-constructed point regression mapping module in sequence through a depth residual block, multi-scale feature down-sampling, global average pooling, and a fully connected layer to predict endpoint offsets.

[0094] The generating unit is configured to generate measurement endpoint coordinates according to the initial bone endpoints and the endpoint offsets.

[0095] In some embodiments, the deployment module 406 includes:

[0096] The end-to-end neural network model is converted into a lightweight inference engine format and deployed in an edge computing or an ultrasound workstation device of a target clinical ultrasound image analysis system, receives original ultrasound frames of real-time fetal ultrasound images through a standardized API interface, and outputs fetal long bone segmentation regions and bone length measurements.

[0097] It should be noted that the foregoing explanation and description of the embodiment of the method for skeleton segmentation and measurement based on fetal four-limb multi-section ultrasound images also applies to the embodiment of the device for skeleton segmentation and measurement based on fetal four-limb multi-section ultrasound images, which will not be described here again.

[0098] The device for skeleton segmentation and measurement based on fetal four-limb multi-section ultrasound images according to the embodiment of the present application has the following beneficial effects:

[0099] (1) The present application can stably extract bone edge features in a complex noise environment by introducing a semantic alignment and adaptive sampling mechanism in the segmentation network, greatly reduces the misclassification and missed classification phenomenon, and significantly improves the Dice and IoU indicators of the segmentation model, ensuring high precision and high consistency of the segmentation result.

[0100] (2) The joint point regression mapping strategy can automatically learn the expert labeling bias, realize accurate positioning of the skeleton endpoint, compared with the traditional edge fitting or manual labeling method, the measurement error is significantly reduced, the measurement result is more close to the actual demand of clinic, and the uncertainty caused by human intervention is reduced;

[0101] (3) The end-to-end lightweight design is adopted, no additional post-processing is needed in the training and inference process, the model inference speed is fast, the resource occupation is low, the model can be seamlessly integrated in the ultrasonic instrument and the edge computing platform, realizes real-time and automatic skeleton segmentation and measurement, greatly improves the clinical work efficiency, reduces the operation burden of medical staff, and is suitable for clinical real-time auxiliary diagnosis scene.

[0102] Figure 5 The structure schematic diagram of the electronic equipment provided by the embodiment of the application is provided. The electronic equipment can include:

[0103] The memory 501, the processor 502 and the computer program stored in the memory 501 and executable on the processor 502.

[0104] The processor 502 implements the skeleton segmentation and measurement method based on the fetal four-limb multi-section ultrasonic image provided in the above embodiment when executing the program.

[0105] Further, the electronic equipment further includes:

[0106] The communication interface 503 is used for communication between the memory 501 and the processor 502.

[0107] The memory 501 is used for storing the computer program executable on the processor 502.

[0108] The memory 501 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0109] If the memory 501, the processor 502 and the communication interface 503 are independently implemented, the communication interface 503, the memory 501 and the processor 502 can be connected with each other through a bus and complete the communication between each other. The bus can be an industry standard architecture (Industry Standard Architecture, abbreviated as ISA) bus, a peripheral component (Peripheral Component, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5Only one bus or only one type of bus can exist, however.

[0110] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete the communication among each other through an internal interface.

[0111] The processor 502 can be a central processing unit (CPU) or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application.

[0112] The embodiments of the present application also provide a computer program product, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the above method for skeleton segmentation and measurement based on a plurality of fetal limb multi-section ultrasound images.

[0113] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method for skeleton segmentation and measurement based on a plurality of fetal limb multi-section ultrasound images.

[0114] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0115] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0116] Any processes or methods described in the flowcharts or otherwise described herein can be understood as representing a module, segment, or portion of code that includes one or N steps for implementing the specified logical functions or processes. The scope of a preferred embodiment of the present application encompasses combinations with additional steps as appropriate, and the steps performed in the manner described or in substantially the same manner but not specifically described in manifestly the same order are to be understood within the scope of the present application.

[0117] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, and can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or N wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical apparatus), and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program can be printed, because the program can be electronically captured, via the optical scan of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in the computer memory.

[0118] It should be understood that portions of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, any of the following technologies known in the art or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0119] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0120] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0121] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs, characterized in that: The following steps are involved: Collecting multi-section ultrasound image data covering the entire process of fetal development, and annotating the multi-section ultrasound image data to construct a segmentation and measurement data set; Introducing semantic alignment skip connection module and feature selection module into the skeleton segmentation neural network framework built in advance with U-Net to obtain the skeleton segmentation neural network; Optimizing the skeleton segmentation neural network framework according to the segmentation and measurement data sets to obtain a skeleton structure segmentation model; Utilizing a pre-built bone length measurement module to receive the segmentation result of the bone structure segmentation model to extract the measurement endpoint coordinates; Constructing an end-to-end neural network architecture based on the bone structure segmentation model and the bone length measurement module, and jointly optimizing and training the end-to-end neural network architecture using a pre-constructed hybrid objective function to obtain an end-to-end neural network model; The end-to-end neural network model is deployed in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output fetal long bone segmentation regions and bone length measurements.

2. The bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs according to claim 1, characterized in that: The collecting of multi-section ultrasound image data covering the entire fetal development process and labeling of the multi-section ultrasound image data to construct a segmentation and measurement training data set includes: Collecting the multi-section ultrasound image data covering the entire process of fetal development, wherein the multi-section ultrasound image data covering the entire process of fetal development includes scanning section images corresponding to the femur, humerus, tibia and fibula, and radius and ulna in the fetus's limbs; The bone region in each scanned section image is segmented and labeled with a pixel-level mask, and the measurement endpoints at both ends of the bone are labeled to construct the segmentation and measurement training dataset.

3. The bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs according to claim 1, characterized in that: The semantic alignment skip connection module and feature selection module are introduced into the skeleton segmentation neural network framework pre-built with U-Net to obtain the skeleton segmentation neural network, including: Building the skeleton segmentation neural network framework based on the classic U-Net structure; Embedding the semantic alignment skip connection module in each level of skip connection of the skeleton segmentation neural network framework to reconstruct the projected features through cross-channel attention and spatial-channel attention mechanisms; A feature selection module is added to the bottleneck layer in the skeleton segmentation neural network framework to filter out noise and amplify skeleton features using a dynamic threshold.

4. The bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs according to claim 1, characterized in that: The skeleton segmentation neural network framework is optimized according to the segmentation and measurement data sets to obtain a skeleton structure segmentation model, including: Setting the classification loss and the region matching loss of the skeleton segmentation neural network as a joint optimization target; Preprocessing the segmentation and measurement data set to obtain a preprocessed segmentation and measurement data set; Based on a staged training approach, the skeleton segmentation neural network framework is optimized using the preprocessed segmentation and measurement data sets until the joint optimization objective converges, thereby obtaining the skeleton structure segmentation model.

5. The bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs according to claim 1, characterized in that: The method of using a pre-built bone length measurement module to receive the segmentation result of the bone structure segmentation model to extract the measurement endpoint coordinates includes: Utilizing a pre-built bone length measurement module to receive the segmentation result of the bone structure segmentation model to extract initial bone endpoints; The segmentation results are processed using a pre-built point regression mapping module through a depth residual block, multi-scale feature downsampling, global average pooling, and a fully connected layer to predict endpoint offsets; The measurement endpoint coordinates are generated according to the initial bone endpoint and the endpoint offset.

6. The bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs according to claim 1, characterized in that: The end-to-end neural network model is deployed in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output fetal long bone segmentation regions and bone length measurements, including: The end-to-end neural network model is converted into a lightweight inference engine format and deployed in the edge computing or ultrasound workstation device of the target clinical ultrasound image analysis system. The original ultrasound frame of the real-time fetal ultrasound image is received through a standardized API interface, and the fetal long bone segmentation area and the bone length measurement value are output.

7. A bone segmentation and measurement device based on multi-section ultrasound images of fetal limbs, characterized in that: include: a labeling module for collecting multi-section ultrasound image data covering the entire process of fetal development and labeling the multi-section ultrasound image data to construct a segmentation and measurement data set; A network construction module is used to introduce a semantic alignment skip connection module and a feature selection module into the skeleton segmentation neural network framework pre-built with U-Net to obtain a skeleton segmentation neural network; A network optimization module is used to optimize the skeleton segmentation neural network framework according to the segmentation and measurement data sets to obtain a skeleton structure segmentation model; An endpoint measurement module, configured to utilize a pre-built bone length measurement module to receive the segmentation results of the bone structure segmentation model to extract measurement endpoint coordinates; a joint optimization module, configured to construct an end-to-end neural network architecture based on the bone structure segmentation model and the bone length measurement module, and to perform joint optimization training on the end-to-end neural network architecture using a pre-built hybrid objective function to obtain an end-to-end neural network model; A deployment module is used to deploy the end-to-end neural network model in a target clinical ultrasound image analysis system to receive real-time fetal ultrasound images and output fetal long bone segmentation regions and bone length measurements.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs as described in any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program / instruction is executed by a processor, the bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the bone segmentation and measurement method based on multi-section ultrasound images of fetal limbs as described in any one of claims 1 to 6.

Images