Method and device for identifying standard section of ultrasonic image, program product, electronic equipment and storage medium

Abstract

The invention provides an ultrasonic image standard section recognition method and device, a program product, electronic equipment and a storage medium, and is applied to the technical field of image processing, the ultrasonic image standard section recognition method comprises the steps that first feature data is acquired, and the first feature data is determined based on continuous multiple frames of to-be-recognized ultrasonic images; the first feature data is input into a standard section recognition network, a standard section target image corresponding to the ultrasonic image to be recognized and output by the standard section recognition network is obtained, the standard section recognition network is a U-shaped network, and jump connection of the standard section recognition network comprises position relation constraints. The standard section recognition network can adopt a U-shaped network, and position relation constraints can be introduced into jump connection of the U-shaped network, so that the reasonability of an organ structure is maintained in the segmentation process, the abnormal spatial relation or shape of a segmentation result is avoided, and the recognition accuracy of the standard section of the ultrasonic image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for recognizing standard sections of ultrasound images, a program product, an electronic device, and a storage medium. Background Art

[0002] In recent years, ultrasound imaging has been increasingly used in clinical diagnosis, especially in the fields of obstetrics and gynecology, cardiology, etc. However, since the quality of ultrasound images is easily affected by factors such as doctor operation and angle, obtaining standard sections has become a major problem in the diagnosis process. Especially in primary medical institutions, insufficient doctor experience may lead to misdiagnosis or missed diagnosis. How to ensure image quality and improve work efficiency has become an urgent problem to be solved. In the existing technology, the accuracy of ultrasound image standard section recognition is low. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method and device for identifying standard sections of ultrasound images, a program product, an electronic device, and a storage medium, so as to solve the technical problem of low accuracy in identifying standard sections of ultrasound images in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a method for identifying standard sections of an ultrasound image, comprising: acquiring first feature data, wherein the first feature data is determined based on a continuous plurality of frames of ultrasound images to be identified; inputting the first feature data into a standard section recognition network to obtain a standard section target image corresponding to the ultrasound image to be identified, output by the standard section recognition network, wherein the standard section recognition network is a U-shaped network, and the jump connections of the standard section recognition network include positional relationship constraints.

[0005] In the above scheme, after obtaining the first feature data, the standard section recognition network can be used to identify the standard section target image of the ultrasound image to be identified based on the above first feature data; wherein the above standard section recognition network can adopt a U-shaped network, and positional relationship constraints can be introduced in the jump connection of the above U-shaped network, so as to maintain the rationality of the organ structure during the segmentation process and avoid abnormal spatial relationships or shapes in the segmentation results, thereby improving the accuracy of standard section recognition of ultrasound images.

[0006] In an optional embodiment, the positional relationship constraint includes the relative position and quantity relationship between key anatomical structures. In the above scheme, the relative position and quantity relationship between key anatomical structures can be introduced into the U-shaped network, thereby constraining the standard section recognition network to consider the reasonable spatial combination of each key structure while extracting it, ensuring the accurate spatial arrangement of the segmented structure, thereby improving the accuracy of standard section recognition of ultrasound images.

[0007] In an optional embodiment, the obtaining of the first feature data includes: obtaining the multiple frames of ultrasound images to be identified; inputting the multiple frames of ultrasound images to be identified into a 3D convolutional neural network to obtain a time series feature map output by the 3D convolutional neural network; performing patch division on the time series feature map to obtain multiple groups of feature image blocks corresponding to the time series feature map; performing Transformer feature extraction on each group of feature image blocks respectively to obtain second feature data corresponding to each group of feature image blocks; and splicing multiple of the second feature data to obtain the first feature data. In the above scheme, firstly, a 3D convolutional neural network can be used to extract the temporal features of the ultrasound image to be identified, and then these features can be passed to the subsequent Patch division module; the Patch division module can divide the temporal feature map into patches of different scales. This division not only retains the spatial information, but also captures the contextual information of different details through multi-scale division, which helps to more comprehensively understand the relationship between structures; next, Transformer feature extraction can be performed on multiple groups of feature image blocks, so that the model can focus on important anatomical structure areas at different scales; finally, by splicing the second feature data at the output layer, a feature representation that includes local features and global structural relationships is formed, providing more accurate input for the subsequent standard section recognition network.

[0008] In an optional embodiment, before inputting the first feature data into the standard section recognition network to obtain the standard section target image corresponding to the ultrasound image to be recognized output by the standard section recognition network, the method further includes: training the model to be trained using the following steps to obtain the standard section recognition network: obtaining a training data set, wherein the training data set includes a sample ultrasound image and a standard section real image corresponding to the sample ultrasound image; determining third feature data based on the sample ultrasound image; inputting the third feature data into the model to be trained to obtain the standard section predicted image corresponding to the sample ultrasound image output by the model to be trained; calculating the total loss value between the standard section predicted image and the standard section real image; optimizing the model to be trained using the total loss value to obtain the standard section recognition network. In the above scheme, before using the standard section recognition network to identify the standard section target image of the ultrasound image to be recognized based on the first feature data, the network to be trained can be trained to obtain the above-mentioned standard section target image.

[0009] In an optional embodiment, the calculating the total loss value between the standard section prediction image and the standard section real image includes: calculating the segmentation loss value between the standard section prediction image and the standard section real image using the following formula:

[0010] L seg=α·Diceloss+β·BCEloss;

[0011] Among them, L seg is the segmentation loss value, Dice loss is the Diss loss value and is used to measure the similarity between the standard section prediction image and the standard section real image, BCE loss is the BCE loss value and is used to measure the difference between the probability distribution of the standard section prediction image and the probability distribution of the standard section real image, α and β are the weights of the Diss loss value and the BCE loss value respectively; and / or, the shape loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0012]

[0013] Among them, L shp is the shape loss value, Intersection is the intersection between the segmented area in the standard section prediction image and the segmented area in the standard section real image, Union is the union between the segmented area in the standard section prediction image and the segmented area in the standard section real image, γ is the weight of the shape loss value; and / or, the position loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0014]

[0015] Among them, L loc is the position loss value, p i and q i are the actual position coordinates and the expected position coordinates of the key anatomical structure, respectively, and δ is the weight of the position loss value; and / or, the fractional loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0016] L scg =ε·(PredictedScore-TargetScore) 2 ;

[0017] Among them, L scris the score loss value, Predicted Score is the score of the predicted image of the standard section, Target Score is the score of the real image of the standard section, and v is the weight of the score loss value. In the above scheme, segmentation loss can be used to combine Dice loss and BCE loss to achieve accurate structure segmentation; shape loss can be used to constrain the shape of key anatomical structures to ensure that the segmentation results conform to the shape characteristics of the real structure; position loss can be used to make the structure in the segmentation result conform to the expected spatial distribution; score loss can be used to score the standard section of the image, and the section with a high score is retained as the standard section.

[0018] In an optional embodiment, the method for identifying standard sections of ultrasound images further includes: inputting the predicted standard section image into a prediction score model to obtain the score of the predicted standard section image output by the prediction score model; and inputting the real standard section image into the prediction score model to obtain the score of the real standard section image output by the prediction score model. In the above scheme, a prediction score model can be constructed to measure the difference between the segmentation quality and the score as an optimization target.

[0019] In a second aspect, an embodiment of the present application provides an apparatus for identifying standard sections of ultrasound images, comprising: an acquisition module for acquiring first feature data, wherein the first feature data is determined based on a continuous plurality of frames of ultrasound images to be identified; and a first input module for inputting the first feature data into a standard section recognition network to obtain a standard section target image output by the standard section recognition network and corresponding to the ultrasound image to be identified, wherein the standard section recognition network is a U-shaped network, and the jump connections of the standard section recognition network include positional relationship constraints.

[0020] In the above scheme, after obtaining the first feature data, the standard section recognition network can be used to identify the standard section target image of the ultrasound image to be identified based on the above first feature data; wherein the above standard section recognition network can adopt a U-shaped network, and positional relationship constraints can be introduced in the jump connection of the above U-shaped network, so as to maintain the rationality of the organ structure during the segmentation process and avoid abnormal spatial relationships or shapes in the segmentation results, thereby improving the accuracy of standard section recognition of ultrasound images.

[0021] In an optional embodiment, the positional relationship constraint includes the relative position and quantity relationship between key anatomical structures. In the above scheme, the relative position and quantity relationship between key anatomical structures can be introduced into the U-shaped network, thereby constraining the standard section recognition network to consider the reasonable spatial combination of each key structure while extracting it, ensuring the accurate spatial arrangement of the segmented structure, thereby improving the accuracy of standard section recognition of ultrasound images.

[0022] In an optional embodiment, the acquisition module is specifically used to: acquire the multiple frames of ultrasound images to be identified; input the multiple frames of ultrasound images to be identified into a 3D convolutional neural network to obtain a time series feature map output by the 3D convolutional neural network; perform patch division on the time series feature map to obtain multiple groups of feature image blocks corresponding to the time series feature map; perform Transformer feature extraction on each group of feature image blocks respectively to obtain second feature data corresponding to each group of feature image blocks; and splice multiple of the second feature data to obtain the first feature data. In the above scheme, firstly, a 3D convolutional neural network can be used to extract the temporal features of the ultrasound image to be identified, and then these features can be passed to the subsequent Patch division module; the Patch division module can divide the temporal feature map into patches of different scales. This division not only retains the spatial information, but also captures the contextual information of different details through multi-scale division, which helps to more comprehensively understand the relationship between structures; next, Transformer feature extraction can be performed on multiple groups of feature image blocks, so that the model can focus on important anatomical structure areas at different scales; finally, by splicing the second feature data at the output layer, a feature representation that includes local features and global structural relationships is formed, providing more accurate input for the subsequent standard section recognition network.

[0023] In an optional embodiment, the device for identifying standard sections of ultrasound images further includes: a training module, which is used to train the model to be trained using the following steps to obtain the standard section recognition network: obtaining a training data set, wherein the training data set includes a sample ultrasound image and a real image of the standard section corresponding to the sample ultrasound image; determining third feature data based on the sample ultrasound image; inputting the third feature data into the model to be trained to obtain a predicted standard section image corresponding to the sample ultrasound image output by the model to be trained; calculating the total loss value between the predicted standard section image and the real standard section image; optimizing the model to be trained using the total loss value to obtain the standard section recognition network. In the above scheme, before using the standard section recognition network to identify the target standard section image of the ultrasound image to be identified based on the first feature data, the network to be trained can be trained to obtain the above-mentioned target standard section image.

[0024] In an optional implementation manner, the training module is specifically used to: calculate the segmentation loss value between the standard section prediction image and the standard section real image using the following formula:

[0025] L sre =α·Dice loss+β·BCE loss;

[0026] Among them, L seg is the segmentation loss value, Dice loss is the Diss loss value and is used to measure the similarity between the standard section prediction image and the standard section real image, BCE loss is the BCE loss value and is used to measure the difference between the probability distribution of the standard section prediction image and the probability distribution of the standard section real image, α and β are the weights of the Diss loss value and the BCE loss value respectively; and / or, the shape loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0027]

[0028] Among them, L shp is the shape loss value, Intersection is the intersection between the segmented area in the standard section prediction image and the segmented area in the standard section real image, Union is the union between the segmented area in the standard section prediction image and the segmented area in the standard section real image, γ is the weight of the shape loss value; and / or, the position loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0029]

[0030] Among them, L loc is the position loss value, p i and q i are the actual position coordinates and the expected position coordinates of the key anatomical structure, respectively, and δ is the weight of the position loss value; and / or, the fractional loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0031] L scr =ε·(PredictedScore-TargetScore) 2 ;

[0032] Among them, L scr is the score loss value, Predicted Score is the score of the predicted image of the standard section, Target Score is the score of the real image of the standard section, and ε is the weight of the score loss value. In the above scheme, segmentation loss can be used to combine Dice loss and BCE loss to achieve accurate structure segmentation; shape loss can be used to constrain the shape of key anatomical structures to ensure that the segmentation results conform to the shape characteristics of the real structure; position loss can be used to make the structure in the segmentation result conform to the expected spatial distribution; score loss can be used to score the standard section of the image, and the section with a high score is retained as the standard section.

[0033] In an optional embodiment, the ultrasound image standard section recognition device further includes: a second input module for inputting the standard section prediction image into a prediction score model to obtain the score of the standard section prediction image output by the prediction score model; and inputting the standard section real image into the prediction score model to obtain the score of the standard section real image output by the prediction score model. In the above scheme, a prediction score model can be constructed to measure the difference between the segmentation quality and the score as an optimization target.

[0034] In a third aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when read and executed by a processor, execute the method for identifying standard sections of ultrasound images as described in the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus; the processor and the memory communicate with each other through the bus; the memory stores computer program instructions that can be executed by the processor, and the processor calls the computer program instructions to execute the method for identifying standard sections of ultrasound images as described in the first aspect.

[0036] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the method for identifying standard sections of ultrasound images as described in the first aspect.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following specifically cites the embodiments of the present application and describes them in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a method for identifying a standard section of an ultrasound image provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a method for identifying standard sections of an ultrasound image provided in an embodiment of the present application;

[0041] Figure 3 A structural block diagram of a device for identifying standard sections of ultrasound images provided in an embodiment of the present application;

[0042] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0044] Please refer to Figure 1 , Figure 1 A flowchart of a method for identifying a standard section of an ultrasound image provided in an embodiment of the present application, the method for identifying a standard section of an ultrasound image may specifically include the following steps:

[0045] Step S101: Acquire first feature data, wherein the first feature data is determined based on a plurality of consecutive frames of ultrasound images to be identified.

[0046] Step S102: Input the first feature data into the standard section recognition network to obtain a standard section target image corresponding to the ultrasound image to be recognized, which is output by the standard section recognition network, wherein the standard section recognition network is a U-shaped network, and the jump connection of the standard section recognition network includes positional relationship constraints.

[0047] Specifically, in the above step S101, the first feature data is determined based on a continuous plurality of frames of ultrasound images to be identified. The plurality of frames of ultrasound images to be identified may be a continuous plurality of frames of images in an ultrasound video. The embodiment of the present application does not limit the number of the ultrasound images to be identified, for example, three frames, four frames, or five frames.

[0048] For example, for a continuous ultrasound video, the first feature data can be determined based on the first, second, and third frames of ultrasound images to be identified, and the standard section can be identified; then the first feature data can be determined based on the second, third, and fourth frames of ultrasound images to be identified, and the standard section can be identified; then the first feature data can be determined based on the third, fourth, and fifth frames of ultrasound images to be identified, and the standard section can be identified; and so on. In this way, it helps to maintain the continuity of key anatomical structures, ensure that the temporal context can be used during segmentation, and strengthen the recognition of structural consistency.

[0049] It should be noted that the embodiments of the present application do not specifically limit the specific implementation method of determining the first feature data based on multiple consecutive frames of ultrasound images to be identified, and those skilled in the art can make appropriate adjustments according to actual conditions. For example, a deep learning network is used to extract the features of the ultrasound image to be identified; or corner point detection is used to extract the features of the ultrasound image to be identified.

[0050] In addition, the specific implementation method for obtaining the first characteristic data is also not specifically limited in the embodiments of the present application, and those skilled in the art may make appropriate adjustments according to actual conditions. For example, the first characteristic data sent by an external device may be received; or the first characteristic data pre-stored locally or in the cloud may be read; or a plurality of continuous frames of ultrasound images to be identified may be obtained, and the first characteristic data may be determined based on the plurality of continuous frames of ultrasound images to be identified, etc.

[0051] In the above step S102, the standard section recognition network is a U-Net. The U-Net adopts a U-shaped structure, including an encoder (downsampling path) and a decoder (upsampling path). The encoder gradually extracts the features of the input image and reduces the spatial resolution, and the decoder restores the feature map to the size of the original input image through upsampling operations, and gradually generates segmentation results. The U-Net introduces a skip connection between the encoder and the decoder, connecting the feature map in the encoder with the corresponding feature map in the decoder to retain the details and semantic information in the original input data.

[0052] In an embodiment of the present application, the jump connection of the above-mentioned standard section recognition network includes positional relationship constraints, which provide spatial relationships and quantitative constraints of key anatomical structures for the features in the jump connection, thereby maintaining the rationality of the organ structure during the segmentation process and avoiding abnormal spatial relationships or shapes in the segmentation results.

[0053] As an implementation method, the above purpose can be achieved by adding a Relation-Location (RL) module to the U-shaped network. The RL module introduces spatial position and relationship constraints, and by encoding the relative position and quantitative relationship between key anatomical structures, embedding them into the network as a set of relationship vectors, constraining the standard section recognition network to consider the reasonable spatial combination of each key anatomical structure while extracting it.

[0054] Specifically, the RL module provides contextual information of key anatomical structures in the skip connections of the U-network. Each skip connection retains feature information at different resolutions during the segmentation process, ensuring that multi-scale information is effectively fused in the segmentation stage.

[0055] It should be noted that the positional relationship can be modeled in advance, and different recognition targets correspond to different positional relationship constraints. For example, the liver anatomical structure requires the model to recognize and maintain the I-shaped distribution relationship, that is, the relative arrangement of the left internal lobe branch, sagittal part, transverse part, lower branch of the left lateral lobe, and upper part of the left lateral lobe.

[0056] In the above step S102, by inputting the first feature data into the standard section recognition network, a standard section target image corresponding to the ultrasound image to be recognized output by the standard section recognition network can be obtained, thereby realizing the recognition of the standard section of the ultrasound image.

[0057] In the above scheme, after obtaining the first feature data, the standard section recognition network can be used to identify the standard section target image of the ultrasound image to be identified based on the above first feature data; wherein the above standard section recognition network can adopt a U-shaped network, and positional relationship constraints can be introduced in the jump connection of the above U-shaped network, so as to maintain the rationality of the organ structure during the segmentation process and avoid abnormal spatial relationships or shapes in the segmentation results, thereby improving the accuracy of standard section recognition of ultrasound images.

[0058] Further, based on the above embodiments, the above positional relationship constraints include relative positions and quantity relationships between key anatomical structures.

[0059] In the above scheme, the relative position and quantity relationship between key anatomical structures can be introduced into the U-shaped network, thereby constraining the standard section recognition network to consider the reasonable spatial combination of each key structure while extracting it, ensuring the accurate spatial arrangement of the segmented structure, thereby improving the accuracy of standard section recognition of ultrasound images.

[0060] Further, based on the above embodiment, the above step S101 may specifically include the following steps:

[0061] Step 1), obtaining multiple frames of ultrasound images to be identified.

[0062] Step 2), multiple frames of ultrasound images to be identified are input into the 3D convolutional neural network to obtain a temporal feature map output by the 3D convolutional neural network.

[0063] Step 3), perform patch division on the temporal feature graph to obtain multiple groups of feature image blocks corresponding to the temporal feature graph.

[0064] Step 4), performing Transformer feature extraction on each group of feature image blocks respectively to obtain second feature data corresponding to each group of feature image blocks.

[0065] Step 5), concatenate multiple second feature data to obtain first feature data.

[0066] Specifically, in the above step 1), the above-mentioned multiple frames of ultrasound images to be identified can be multiple groups of continuous multi-frame images in the ultrasound video. The embodiment of the present application does not limit the number of the above-mentioned groups of ultrasound images to be identified, for example: three frames, four frames or five frames, etc.

[0067] It should be noted that the specific implementation method of acquiring multiple frames of ultrasound images to be identified is also not specifically limited in the embodiments of the present application, and those skilled in the art may make appropriate adjustments according to actual conditions. For example, multiple frames of ultrasound images to be identified sent by an external device may be received; or multiple frames of ultrasound images to be identified pre-stored locally or in the cloud may be read; or multiple frames of ultrasound images to be identified may be collected in real time, etc.

[0068] In step 2), 3D Convolutional Neural Networks (C3D) is very suitable for processing time-series video data and can extract features from three-dimensional space. The time-series convolution operation of this network can retain the time-series information of consecutive frames. Specifically, C3D first performs time-series convolution and pooling on the three frames of input video to generate time-series features, and then passes these features to the subsequent patch division module.

[0069] In the above step 3), in order to further refine the structural information, different scales of patch division (for example: 4Patch, 9Patch, ..., 64Patch) can be performed on the temporal feature map extracted in the above step 2). This division not only retains the spatial information, but also captures the contextual information of different details through multi-scale division, which helps to more comprehensively understand the relationship between structures.

[0070] Among them, through different patch sizes, the model can better capture local structures and global relationships. Small-sized patches (such as 64 patches) focus on details, while large-sized patches (such as 4 patches) help capture larger-scale anatomical structures. Each patch is divided into an independent sub-region to form a smaller feature block.

[0071] In the above step 4), each group of feature image blocks can be input into the Transformer model, and their mutual relations can be extracted through the self-attention mechanism. Among them, the Transformer uses self-attention to assign weights to each patch, pay attention to the importance of different patches, and extract the interactive features of each patch at multiple levels, so as to obtain the relationship information of spatial positions.

[0072] As an implementation, the Transformer model can include multi-head self-attention, which assigns different weights to each patch, allowing the model to focus on important anatomical regions at different scales. For example, the model can pay more attention to patches containing key anatomical information and ignore other redundant backgrounds.

[0073] In the above step 5), multiple second feature data can be concatenated at the output layer to form a global feature representation, which includes local features and global structural relationships, providing more accurate input for the subsequent standard section recognition network.

[0074] In the above scheme, firstly, a 3D convolutional neural network can be used to extract the temporal features of the ultrasound image to be identified, and then these features can be passed to the subsequent Patch division module; the Patch division module can divide the temporal feature map into patches of different scales. This division not only retains the spatial information, but also captures the contextual information of different details through multi-scale division, which helps to more comprehensively understand the relationship between structures; next, Transformer feature extraction can be performed on multiple groups of feature image blocks, so that the model can focus on important anatomical structure areas at different scales; finally, by splicing the second feature data at the output layer, a feature representation that includes local features and global structural relationships is formed, providing more accurate input for the subsequent standard section recognition network.

[0075] Further, based on the above embodiment, before the above step S102, the method for recognizing standard sections of ultrasound images provided in the embodiment of the present application may further include the following steps:

[0076] Use the following steps to train the model to be trained to obtain the standard section recognition network:

[0077] Step 1) Obtain a training data set, wherein the training data set includes a sample ultrasound image and a standard section real image corresponding to the sample ultrasound image.

[0078] Step 2), determining third characteristic data based on the sample ultrasound image.

[0079] Step 3), input the third characteristic data into the model to be trained, and obtain the standard section prediction image corresponding to the sample ultrasound image output by the model to be trained.

[0080] Step 4), calculate the total loss value between the standard section prediction image and the standard section real image; use the total loss value to optimize the training model to obtain the standard section recognition network.

[0081] In the above scheme, before using the standard section recognition network to recognize the standard section target image of the ultrasound image to be recognized based on the first feature data, the network to be trained can be trained to obtain the standard section target image.

[0082] Further, based on the above embodiment, as a first implementation manner, the above step of calculating the total loss value between the standard section prediction image and the standard section real image may specifically include the following steps:

[0083] The segmentation loss between the standard section prediction image and the standard section real image is calculated using the following formula:

[0084] L seg =α·Dice loss+β·BCE loss;

[0085] Among them, L seg is the segmentation loss value, Dice loss is the Diss loss value and is used to measure the similarity between the standard section prediction image and the standard section real image, BCE loss is the binary cross entropy (BCE) loss value and is used to measure the difference between the probability distribution of the standard section prediction image and the probability distribution of the standard section real image, α and β are the weights of the Diss loss value and the BCE loss value, respectively.

[0086] Specifically, the segmentation loss is used to measure the overlap between the segmentation result and the true annotation, and a combination of Dice Loss and BCE Loss can be used; among them, Dice Loss increases the sensitivity of the model to the segmented area, while BCE Loss is used to deal with the imbalance between the foreground and background areas.

[0087] Dice Loss is a loss function based on the Dice coefficient, which is used to measure the similarity between the predicted segmentation result and the true label. As an implementation method, the Diss loss value can be calculated using the following formula:

[0088] Dice loss = 1-Dice(A,B);

[0089] Where A represents the pixel set of the predicted result, B represents the pixel set of the true label, Dice(A,B) is the Dice coefficient, and its calculation formula is Dice(A,B)=2|A∩B| / (|A|+|B|), where |A∩B| represents the number of intersection pixels of the predicted result and the true label, and |A| and |B| represent the number of pixels of the predicted result and the true label, respectively.

[0090] BCE Loss measures the difference between the probability distribution predicted by the model and the probability distribution of the true label. As an implementation method, the BCE loss value can be calculated using the following formula:

[0091]

[0092] Among them, y i represents the i-th pixel value of the true label (0 or 1), It represents the probability value predicted by the model that the i-th pixel belongs to the target category, and N represents the total number of pixels.

[0093] In the above scheme, segmentation loss can be used, combined with Dice loss and BCE loss to achieve accurate structure segmentation.

[0094] Further, based on the above embodiment, as a second implementation manner, the above step of calculating the total loss value between the standard section prediction image and the standard section real image may specifically include the following steps:

[0095] The shape loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0096]

[0097] Among them, L shp is the shape loss value, Intersection is the intersection between the segmented area in the standard section prediction image and the segmented area in the standard section real image, Union is the union between the segmented area in the standard section prediction image and the segmented area in the standard section real image, and γ is the weight of the shape loss value.

[0098] Specifically, the shape loss constrains the shape of the key anatomical structure to ensure that the segmentation result conforms to the shape characteristics of the real structure. As an implementation method, the boundary distance or intersection over union (IoU) calculation can be used, or the Hausdorff distance can be used to calculate the consistency of the boundary.

[0099] Among them, Intersection refers to the intersection between the predicted segmentation area and the true segmentation area (Ground Truth), Intersection = ∑ (Pred ∩ GT), ∩ represents the pixel-by-pixel bitwise AND operation, Union refers to the union of the predicted segmentation area and the true segmentation area, Union = ∑ (Pred ∪ GT), ∪ represents the pixel-by-pixel bitwise OR operation.

[0100] In the above scheme, shape loss can be used to constrain the shape of key anatomical structures to ensure that the segmentation results conform to the shape characteristics of the real structure.

[0101] Further, based on the above embodiment, as a third implementation manner, the above step of calculating the total loss value between the standard section prediction image and the standard section real image may specifically include the following steps:

[0102] The position loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0103]

[0104] Among them, L loc is the location loss value, p i and q i are the actual position coordinates and expected position coordinates of the key anatomical structure, respectively, and δ is the weight of the position loss value.

[0105] Specifically, the position loss calculates the relative position error of the key anatomical structure so that the structure in the segmentation result conforms to the expected spatial distribution. As an implementation method, the Euclidean distance can be used to calculate the difference between the actual and target positions.

[0106] In the above scheme, position loss can be used to make the structure in the segmentation result conform to the expected spatial distribution.

[0107] Further, based on the above embodiment, as a fourth implementation manner, the above step of calculating the total loss value between the standard section prediction image and the standard section real image may specifically include the following steps:

[0108] The score loss between the standard section prediction image and the standard section true image is calculated using the following formula:

[0109] L scr =ε·(Predicted Score-Target Score) 2 ;

[0110] Among them, L scr is the score loss value, Predicted Score is the score of the standard section predicted image, Target Score is the score of the standard section real image, and ε is the weight of the score loss value.

[0111] Specifically, the score loss is scored according to the standard slices of the image, and the slices with high scores are retained as standard slices.

[0112] In the above scheme, a score loss can be used to score the standard slices of the image, and the slices with high scores are retained as standard slices.

[0113] Further, based on the above embodiment, the model to be trained can be trained based on one or more of the above four loss values. Taking the training of the model to be trained based on the above four loss values ​​together as an example, the overall loss function can be expressed as:

[0114] L loc =α·L seg +β·L shp +γ·L loc +δ·L scr .

[0115] Further, based on the above embodiment, the method for identifying standard sections of ultrasound images provided in the embodiment of the present application may also include the following steps:

[0116] The standard section prediction image is input into the prediction score model to obtain the score of the standard section prediction image output by the prediction score model; and the standard section real image is input into the prediction score model to obtain the score of the standard section real image output by the prediction score model.

[0117] In the above scheme, a prediction score model can be constructed to measure the difference between the segmentation quality and the score as the optimization goal.

[0118] Please refer to Figure 2 , Figure 2 A schematic diagram of a method for identifying a standard section of an ultrasound image provided in an embodiment of the present application. Figure 2 As shown, an ultrasound video is obtained; three consecutive frames of ultrasound images are extracted from the ultrasound video; a time series feature map is obtained by inputting the image into C3D; the time series feature map is divided into patches, including 2*2Patch, 3*3Patch, ..., 8*8Patch; Transformer feature extraction is performed on the feature image blocks respectively, and the first feature data is obtained by splicing; the first feature data is input into U-Net, wherein an RL module is introduced into the jump connection of U-Net; finally, the model is trained based on Segmentation Loss, Shape Loss, Location Loss, and Score Loss; a standard section image is output to complete the recognition of the standard section of the ultrasound image.

[0119] Please refer to Figure 3 , Figure 3 A structural block diagram of an ultrasound image standard section recognition device provided in an embodiment of the present application, the ultrasound image standard section recognition device 300 specifically includes: an acquisition module 301, used to acquire first feature data, wherein the first feature data is determined based on a continuous plurality of frames of ultrasound images to be recognized; a first input module 302, used to input the first feature data into a standard section recognition network to obtain a standard section target image corresponding to the ultrasound image to be recognized output by the standard section recognition network, wherein the standard section recognition network is a U-shaped network, and the jump connections of the standard section recognition network include positional relationship constraints.

[0120] In the above scheme, after obtaining the first feature data, the standard section recognition network can be used to identify the standard section target image of the ultrasound image to be identified based on the above first feature data; wherein the above standard section recognition network can adopt a U-shaped network, and positional relationship constraints can be introduced in the jump connection of the above U-shaped network, so as to maintain the rationality of the organ structure during the segmentation process and avoid abnormal spatial relationships or shapes in the segmentation results, thereby improving the accuracy of standard section recognition of ultrasound images.

[0121] Further, based on the above embodiment, the positional relationship constraints include the relative position and quantity relationship between key anatomical structures.

[0122] In the above scheme, the relative position and quantity relationship between key anatomical structures can be introduced into the U-shaped network, thereby constraining the standard section recognition network to consider the reasonable spatial combination of each key structure while extracting it, ensuring the accurate spatial arrangement of the segmented structure, thereby improving the accuracy of standard section recognition of ultrasound images.

[0123] Furthermore, based on the above embodiments, the acquisition module 301 is specifically used to: acquire the multiple frames of ultrasound images to be identified; input the multiple frames of ultrasound images to be identified into a 3D convolutional neural network to obtain a time series feature map output by the 3D convolutional neural network; perform patch division on the time series feature map to obtain multiple groups of feature image blocks corresponding to the time series feature map; perform Transformer feature extraction on each group of feature image blocks respectively to obtain second feature data corresponding to each group of feature image blocks; and splice multiple of the second feature data to obtain the first feature data.

[0124] In the above scheme, firstly, a 3D convolutional neural network can be used to extract the temporal features of the ultrasound image to be identified, and then these features can be passed to the subsequent Patch division module; the Patch division module can divide the temporal feature map into patches of different scales. This division not only retains the spatial information, but also captures the contextual information of different details through multi-scale division, which helps to more comprehensively understand the relationship between structures; next, Transformer feature extraction can be performed on multiple groups of feature image blocks, so that the model can focus on important anatomical structure areas at different scales; finally, by splicing the second feature data at the output layer, a feature representation that includes local features and global structural relationships is formed, providing more accurate input for the subsequent standard section recognition network.

[0125] Furthermore, on the basis of the above-mentioned embodiment, the apparatus 300 for identifying standard sections of ultrasound images also includes: a training module, which is used to train the model to be trained using the following steps to obtain the standard section recognition network: obtaining a training data set, wherein the training data set includes a sample ultrasound image and a real image of the standard section corresponding to the sample ultrasound image; determining third feature data based on the sample ultrasound image; inputting the third feature data into the model to be trained to obtain a predicted standard section image corresponding to the sample ultrasound image output by the model to be trained; calculating a total loss value between the predicted standard section image and the real standard section image; and optimizing the model to be trained using the total loss value to obtain the standard section recognition network.

[0126] In the above scheme, before using the standard section recognition network to recognize the standard section target image of the ultrasound image to be recognized based on the first feature data, the network to be trained can be trained to obtain the standard section target image.

[0127] Further, based on the above embodiment, the training module is specifically used to calculate the segmentation loss value between the standard section prediction image and the standard section real image using the following formula:

[0128] L seg =α·Dice loss+β·BCE loss;

[0129] Among them, L seg is the segmentation loss value, Dice loss is the Diss loss value and is used to measure the similarity between the standard section prediction image and the standard section real image, BCE loss is the BCE loss value and is used to measure the difference between the probability distribution of the standard section prediction image and the probability distribution of the standard section real image, α and β are the weights of the Diss loss value and the BCE loss value respectively; and / or, the shape loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0130]

[0131] Among them, L shp is the shape loss value, Intersection is the intersection between the segmented area in the standard section prediction image and the segmented area in the standard section real image, Union is the union between the segmented area in the standard section prediction image and the segmented area in the standard section real image, γ is the weight of the shape loss value; and / or, the position loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0132]

[0133] Among them, L loc is the position loss value, p i and q i are the actual position coordinates and the expected position coordinates of the key anatomical structure, respectively, and δ is the weight of the position loss value; and / or, the fractional loss value between the standard section prediction image and the standard section real image is calculated using the following formula:

[0134] L scr =ε·(Predicted Score-Target Score) 2 ;

[0135] Among them, L scr is the score loss value, Predicted Score is the score of the standard section predicted image, Target Score is the score of the standard section real image, and ε is the weight of the score loss value.

[0136] In the above scheme, segmentation loss can be used in combination with Dice loss and BCE loss to achieve accurate structure segmentation; shape loss can be used to constrain the shape of key anatomical structures to ensure that the segmentation results conform to the shape characteristics of the real structure; position loss can be used to make the structure in the segmentation result conform to the expected spatial distribution; score loss can be used to score the standard sections of the image, and the sections with high scores are retained as standard sections.

[0137] Furthermore, based on the above embodiment, the ultrasound image standard section recognition device 300 also includes: a second input module, used to input the standard section prediction image into the prediction score model to obtain the score of the standard section prediction image output by the prediction score model; and, input the standard section real image into the prediction score model to obtain the score of the standard section real image output by the prediction score model.

[0138] In the above scheme, a prediction score model can be constructed to measure the difference between the segmentation quality and the score as the optimization goal.

[0139] Please refer to Figure 4 , Figure 4The structural block diagram of an electronic device provided in an embodiment of the present application, the electronic device 400 includes: at least one processor 401, at least one communication interface 402, at least one memory 403 and at least one communication bus 404. Among them, the communication bus 404 is used to realize the direct connection and communication of these components, the communication interface 402 is used to communicate signaling or data with other node devices, and the memory 403 stores machine-readable instructions executable by the processor 401. When the electronic device 400 is running, the processor 401 communicates with the memory 403 through the communication bus 404, and the machine-readable instructions are called by the processor 401 to execute the above-mentioned ultrasound image standard section recognition method.

[0140] For example, the processor 401 of the embodiment of the present application reads a computer program from the memory 403 through the communication bus 404 and executes the computer program to implement the following method: obtaining first feature data, wherein the first feature data is determined based on a continuous plurality of frames of ultrasound images to be identified; inputting the first feature data into a standard section recognition network to obtain a standard section target image corresponding to the ultrasound image to be identified output by the standard section recognition network, wherein the standard section recognition network is a U-shaped network, and the jump connections of the standard section recognition network include positional relationship constraints.

[0141] Among them, the processor 401 includes one or more, which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 401 can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; it can also be a dedicated processor, including a neural network processor (NPU), a graphics processor (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. In addition, when there are multiple processors 401, some of them can be general-purpose processors and the other part can be dedicated processors.

[0142] The memory 403 includes one or more, which may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0143] Understandably, Figure 4 The structure shown is for illustration only. The electronic device 400 may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown. Figure 4 Each component shown in can be implemented by hardware, software or a combination thereof. In the embodiment of the present application, the electronic device 400 can be, but not limited to, a physical device such as a desktop computer, a laptop computer, a smart phone, a smart wearable device, a vehicle-mounted device, etc., and can also be a virtual device such as a virtual machine. In addition, the electronic device 400 is not necessarily a single device, but can also be a combination of multiple devices, such as a server cluster, etc.

[0144] The embodiment of the present application also provides a computer program product, including a computer program stored on a computer-readable storage medium, the computer program including computer program instructions, when the computer program instructions are executed by the computer, the computer can execute the steps of the method for recognizing standard sections of ultrasound images in the above embodiment, for example including: Step S101: Acquire first feature data, wherein the first feature data is determined based on a continuous plurality of frames of ultrasound images to be recognized. Step S102: Input the first feature data into a standard section recognition network, and obtain a standard section target image corresponding to the ultrasound image to be recognized output by the standard section recognition network, wherein the standard section recognition network is a U-shaped network, and the jump connection of the standard section recognition network includes positional relationship constraints.

[0145] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the method for identifying standard sections of ultrasound images described in the aforementioned method embodiment.

[0146] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0147] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0149] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM) random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0150] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0151] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying standard sections of an ultrasound image, characterized in that: include: Acquiring first feature data, wherein the first feature data is determined based on a plurality of consecutive frames of ultrasound images to be identified; The first feature data is input into a standard section recognition network to obtain a standard section target image corresponding to the ultrasound image to be recognized, which is output by the standard section recognition network, wherein the standard section recognition network is a U-shaped network, and the jump connection of the standard section recognition network includes positional relationship constraints.

2. The method for recognizing standard sections of ultrasound images according to claim 1, characterized in that: The positional relationship constraints include the relative positions and quantitative relationships between key anatomical structures.

3. The method for recognizing standard sections of ultrasound images according to claim 1, characterized in that: The obtaining of the first characteristic data comprises: Acquiring the multiple frames of ultrasound images to be identified; Inputting the multiple frames of ultrasound images to be identified into a 3D convolutional neural network to obtain a temporal feature graph output by the 3D convolutional neural network; Patch-dividing the time series feature graph to obtain multiple groups of feature image blocks corresponding to the time series feature graph; Performing Transformer feature extraction on each group of feature image blocks respectively to obtain second feature data corresponding to each group of feature image blocks; The first feature data is obtained by concatenating a plurality of the second feature data.

4. The method for recognizing standard sections of ultrasound images according to any one of claims 1 to 3, characterized in that: Before inputting the first feature data into the standard section recognition network to obtain the standard section target image output by the standard section recognition network and corresponding to the ultrasound image to be recognized, the method further includes: The model to be trained is trained using the following steps to obtain the standard section recognition network: Acquire a training data set, wherein the training data set includes a sample ultrasound image and a standard section real image corresponding to the sample ultrasound image; determining third feature data based on the sample ultrasound image; Inputting the third characteristic data into the model to be trained to obtain a standard section prediction image corresponding to the sample ultrasound image output by the model to be trained; Calculating a total loss value between the standard section prediction image and the standard section real image; The model to be trained is optimized using the total loss value to obtain the standard section recognition network.

5. The method for recognizing standard sections of ultrasound images according to claim 4, characterized in that: The calculating the total loss value between the standard section prediction image and the standard section real image includes: The segmentation loss value between the standard section prediction image and the standard section real image is calculated using the following formula: L seg =α·Dice loss+β·BCE loss; Among them, L seg is the segmentation loss value, Dice loss is the Diss loss value and is used to measure the similarity between the standard section prediction image and the standard section real image, BCE loss is the BCE loss value and is used to measure the difference between the probability distribution of the standard section prediction image and the probability distribution of the standard section real image, α and β are the weights of the Diss loss value and the BCE loss value respectively; and / or, The shape loss value between the standard section prediction image and the standard section real image is calculated using the following formula: Among them, L shp is the shape loss value, Intersection is the intersection between the segmented area in the standard section prediction image and the segmented area in the standard section real image, Union is the union between the segmented area in the standard section prediction image and the segmented area in the standard section real image, and γ is the weight of the shape loss value; and / or, The position loss value between the standard section prediction image and the standard section real image is calculated using the following formula: Among them, L loc is the position loss value, p i and q i are the actual position coordinates and the expected position coordinates of the key anatomical structure, respectively, and δ is the weight of the position loss value; and / or, The score loss value between the standard section prediction image and the standard section real image is calculated using the following formula: L scr =ε·(Predicted Score-Target Score) 2 ; Among them, L scr is the score loss value, Predicted Score is the score of the standard section predicted image, Target Score is the score of the standard section real image, and ε is the weight of the score loss value.

6. The method for recognizing standard sections of ultrasound images according to claim 4, characterized in that: Also includes: Inputting the standard section prediction image into a prediction score model to obtain a score of the standard section prediction image output by the prediction score model; And, inputting the standard section real image into the prediction score model to obtain the score of the standard section real image output by the prediction score model.

7. A device for identifying standard sections of ultrasound images, characterized in that: include: An acquisition module, configured to acquire first feature data, wherein the first feature data is determined based on a plurality of consecutive frames of ultrasound images to be identified; The first input module is used to input the first feature data into a standard section recognition network to obtain a standard section target image output by the standard section recognition network and corresponding to the ultrasound image to be recognized, wherein the standard section recognition network is a U-shaped network, and the jump connection of the standard section recognition network includes positional relationship constraints.

8. A computer program product, characterized in that The method comprises computer program instructions, which, when read and executed by a processor, executes the method for recognizing standard sections of ultrasound images as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions that can be executed by the processor, and the processor calls the computer program instructions to execute the method for identifying standard sections of ultrasound images as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a computer, the computer is caused to execute the method for recognizing standard sections of ultrasound images according to any one of claims 1 to 6.

Images