An ultrasonic nodule segmentation method based on cloud computing and a portable terminal

Abstract

The present application relates to the technical field of ultrasonic medical detection, and aims to provide an ultrasonic nodule segmentation method based on cloud computing and a portable terminal.The method comprises the following steps: taking a sufficient amount of ultrasonic videos with nodule pathological images, and obtaining a training data set after cutting and labeling; deploying a lightweight convolutional neural network on a local portable terminal, deploying a deep convolutional neural network on a cloud server, and constructing a dual-network ultrasonic nodule segmentation system; using the training data set to jointly train the two neural networks, inputting a clinical detection ultrasonic video to be identified into the trained dual-network ultrasonic nodule segmentation system frame by frame, and performing joint segmentation and extraction.The present application uses dual-network joint training inference, can achieve real-time video content analysis while achieving optimal algorithm performance, greatly reduces the requirement of data transmission on the communication network compared with the traditional cloud ultrasonic technology, quickly and efficiently obtains the results of nodule segmentation and feature extraction, and truly realizes zero delay.

Description

Technical Field

[0001] This invention relates to ultrasound medical detection technology, and in particular to an ultrasound nodule segmentation method based on cloud computing and portable terminals. Background Technology

[0002] Existing technology has already enabled collaborative interactive high-order imaging, namely cloud ultrasound technology, by deploying cloud servers and local terminals on the basis of portable ultrasound equipment. This collaborative interactive high-order imaging is achieved by deploying cloud servers and local terminals. Ultrasound image data received by the ultrasound transceiver terminal is transmitted to the cloud via the network, where the ultrasound data undergoes secondary processing before being returned to the interactive terminal for display.

[0003] This technology can maximize the use of cloud computing resources, ensuring algorithm performance while freeing the ultrasound system from limitations on portability, thus eliminating the contradiction between enhanced computing power of the ultrasound system and power consumption and heat dissipation of portable devices.

[0004] However, this technology still has limitations: due to network transmission limitations during data transmission for cloud computing, real-time analysis will suffer from latency issues, preventing the analysis from proceeding smoothly. Therefore, this technology can only be used for historical data review and analysis, and cannot perform real-time analysis based on ultrasound data from a local terminal. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an ultrasonic nodule segmentation method based on cloud computing and portable terminals.

[0006] To solve the above-mentioned technical problems, the solution of the present invention is:

[0007] A method for ultrasound nodule segmentation based on cloud computing and portable terminals is provided, comprising the following steps:

[0008] (1) Take a sufficient amount of ultrasound videos with pathological images of nodules, and after cropping and labeling, obtain a training dataset;

[0009] (2) Deploy a lightweight convolutional neural network on a local portable terminal and a deep convolutional neural network on a cloud server to build a dual-network ultrasound nodule segmentation system;

[0010] (3) The two neural networks are jointly trained using the training dataset. The training process includes: the local lightweight convolutional neural network first performs nodule segmentation and feature extraction on some frame images in the video, and then transmits the feature data and the remaining frame images to the cloud server through the network; the deep convolutional neural network performs nodule segmentation and feature extraction on the remaining frame images, and then splices the two extracted features.

[0011] (4) Input the clinical ultrasound video to be identified frame by frame into the trained dual-network ultrasound nodule segmentation system to perform joint segmentation and extraction: The local lightweight convolutional neural network first performs nodule segmentation and feature extraction on some frames of the video, and then transmits the feature data and the remaining frame images to the cloud server through the network; the deep convolutional neural network performs nodule segmentation and feature extraction on the remaining frame images, and then splices the two extracted features; finally, a report including all nodule segmentation feature results is output.

[0012] As a preferred embodiment of the present invention, when creating the training dataset, frame splitting is first performed to mark the effective segments and ultrasound region information in the frame images; then, the original images are cropped according to the ultrasound region information, and nodules in the ultrasound region images within the effective segments are labeled according to clinical medical judgment standards; in the training dataset, a mask binary image of the same size as the ultrasound region image of the nodule is generated for each nodule, and the area with non-zero pixels in the image indicates that there is a nodule at the corresponding position of the ultrasound region image, and vice versa.

[0013] As a preferred embodiment of the present invention, each case is an independent ultrasound video; when performing joint training or joint segmentation and extraction, a nodule segmentation feature is jointly output every 9 frames of ultrasound area images, with each ultrasound video as a unit; wherein, the first 3 frames are extracted by a local lightweight convolutional neural network, and the last 6 frames are extracted by a cloud-based deep convolutional neural network.

[0014] When there are fewer than 9 frames of video ultrasound area images, the first 3 frames are used to extract features by a local lightweight convolutional neural network, and the remaining frames are used to extract features by a deep convolutional neural network in the cloud.

[0015] When there are fewer than 3 frames of ultrasound images in the video, a lightweight convolutional neural network is used to segment and extract all images, which are then used as the final output features of the system.

[0016] As a preferred embodiment of the present invention, in the dual-network nodal segmentation system, the loss function expression is set as follows:

[0017]

[0018] in, This represents the i-th ultrasound image in the t-th image sequence; Indicates and The corresponding mask output; θ represents the predicted output of the lightweight convolutional neural network, and θ represents the network parameters of the lightweight convolutional neural network. α represents the predicted output of the deep convolutional neural network, t represents the batch of videos from 1 to b, b represents the number of videos in the batch, m represents the number of images input to the lightweight convolutional neural network, i represents the input image sequence from 1 to k, and k represents the number of images in the input image sequence.

[0019] As a preferred embodiment of the present invention, the local portable terminal refers to a portable ultrasound device capable of deploying a lightweight convolutional neural network and having computing and running capabilities, or a portable computing device connected to the ultrasound device; the portable computing device refers to a tablet computer, smartphone, laptop computer, or PC host.

[0020] Description of the invention principle:

[0021] When using conventional cloud ultrasound technology for nodule segmentation and feature extraction, all ultrasound video image data needs to be transmitted to the cloud via the network, which requires a very high network data transmission speed.

[0022] This invention innovatively proposes constructing a lightweight convolutional neural network on a local terminal. This network is used to first segment nodules and extract features from a portion of the ultrasound video image. Then, the extracted feature data, along with the remaining unprocessed image frames, is transmitted to the cloud. The cloud then performs further segmentation and feature extraction on the remaining ultrasound images before concatenating the two feature sets. Since a portion of the video image does not require network transmission, but instead transmits the extracted feature data, the pressure on network transmission is significantly reduced. Simultaneously, because the nodule segmentation calculation is performed directly locally, the overall video processing time is greatly shortened.

[0023] The segmentation of medical ultrasound videos is performed in two parts: the first three frames are segmented by a local computing device, while the remaining frames are segmented by a cloud server. These two processes are performed in parallel. Based on this principle, this invention enables real-time segmentation of medical ultrasound videos. The first few frames are processed rapidly by a local lightweight network, while the remaining frames are processed simultaneously by a large cloud network. This parallel processing combines the final results, achieving zero latency.

[0024] If all processing is done by cloud servers, there are two drawbacks: (1) high network transmission speed is required; (2) at the beginning of the video, the cloud needs to wait for data transmission and processing time, and even the fastest network speed cannot achieve real-time processing; if all processing is done locally, it is difficult to deploy a large network on the computing devices of each ultrasound device, which limits the prediction accuracy.

[0025] Compared with the prior art, the technical advantages of the present invention are:

[0026] 1. This invention deploys a lightweight neural network on a local portable terminal and a high-performance, high-accuracy deep convolutional neural network on a cloud server. By using the dual networks for joint training and inference, it can achieve the goal of real-time analysis of video content while optimizing algorithm performance.

[0027] 2. Compared with traditional cloud ultrasound technology, this invention greatly reduces the requirements of communication networks for data transmission, and can quickly and efficiently obtain the results of nodule segmentation and feature extraction, truly achieving zero latency. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the operation of the present invention.

[0029] Figure 2 This is a schematic diagram of an ultrasound image crop.

[0030] Figure 3 This is a schematic diagram of the nodules in the ultrasound image (left) and the corresponding binary image of the mask (right).

[0031] Figure 4 This is a diagram of the dual-network joint training (and inference) structure. Detailed Implementation

[0032] The implementation process of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] The ultrasonic nodule segmentation method based on cloud computing and portable terminals described in this invention has the following process: Figure 1 As shown, the specific steps include:

[0034] 1. Create a training dataset

[0035] A sufficient number of ultrasound videos containing pathological images of nodules were obtained from hospitals or healthcare institutions, and after cropping and annotation, a training dataset was obtained. Specifically, this included:

[0036] like Figure 2 As shown, the ultrasound video is first split into frames, and the valid segments and ultrasound region information are marked. The valid segment refers to the continuous valid ultrasound frames after removing invalid frames such as black frames and long static frames. The ultrasound region information refers to the upper left corner coordinates (x0, y0) and the lower right corner coordinates (x1, y1) of the ultrasound image. Then, the original image is cropped according to the ultrasound region information, that is, the region from (x0, y0) to (x1, y1) of the original image is taken as the ultrasound region image.

[0037] Furthermore, professional physicians labeled the nodules in the ultrasound images within the effective segment according to clinical medical judgment criteria. The labeled ultrasound images were then masked and binarized to create a training dataset. In the training dataset, a masked binary image of the same size as the ultrasound image was generated for each nodule (e.g., ...). Figure 3 (As shown). In the mask binary image, regions with non-zero pixels indicate the presence of nodules at the corresponding locations in the ultrasound image, while regions with zero pixels indicate the absence of nodules.

[0038] 2. Construction of a dual-network ultrasound nodule segmentation system

[0039] The dual-network ultrasonic nodule segmentation system of this invention refers to deploying a lightweight convolutional neural network on a local portable terminal and a deep convolutional neural network on a cloud server, with the two neural networks able to work in coordination.

[0040] The local portable terminal refers to a portable computing device that is connected to ordinary ultrasound equipment, capable of deploying a lightweight convolutional neural network and having computing and running capabilities; the portable computing device refers to a tablet computer, smartphone, laptop computer or PC host.

[0041] In a specific implementation of this invention, Mobilenet is selected as the local lightweight convolutional neural network and deployed on a local portable terminal, while EfficientNet-B6 is selected as the deep learning convolutional neural network and deployed on a cloud server.

[0042] The Mobilenet network consists of nine depthwise convolutions (forming the network backbone) and one convolutional layer with a kernel size of 3×3 and a stride of 2. Each depthwise convolution can be decomposed into a depthwise convolution (Conv dw) and a 1×1 convolution.

[0043] The EfficientNet-B6 network consists of seven Mobile Inverted Bottleneck (MBConv) convolutional modules and a convolutional layer with a kernel size of 3×3 and a stride of 2. Each MBConv module consists of two regular convolutions, one depthwise separable convolution, and one attention module. The network backbone is composed of the MBConv modules.

[0044] Both Mobilenet and EfficientNet-B6 utilize mature technologies and can be replaced with other lightweight networks of similar scale to achieve the same effect. Since this is not the core innovation of this invention, it will not be described in detail.

[0045] In a dual-network nodal segmentation system, the loss function is defined as follows:

[0046]

[0047] in, This represents the i-th ultrasound image in the t-th image sequence; Indicates and The corresponding mask output; This represents the predicted output of the Mobilenet network, where θ represents the network parameters of the Mobilenet network. α represents the predicted output of the EfficientNet-B6 network, t represents the batch video from 1 to b, b represents the number of batch videos, m represents the number of images in the input MobileNet network, i represents the input image sequence from 1 to k, and k represents the number of images in the input image sequence.

[0048] 3. Dual-network joint training

[0049] Although the lightweight convolutional neural network MobileNet is deployed locally and the deep convolutional neural network EfficientNet-B6 is deployed on the server, both neural networks need to work together during training and practical use (inference) to jointly segment and extract features from ultrasound images. The MobileNet network first performs nodule segmentation and feature extraction on a subset of frames from the video, then transmits the feature data and the remaining frames to the cloud server via the network. The EfficientNet-B6 network then performs nodule segmentation and feature extraction on the remaining frames, and concatenates the extracted features from both sets of frames. The final output is a report including all nodule segmentation feature results.

[0050] A specific example is as follows Figure 4 As shown, its operation is as follows:

[0051] (1) Each case is an independent ultrasound video; with an ultrasound video as a unit, every 9 frames of ultrasound area images are jointly output as a nodule segmentation feature; among them, the first 3 frames are extracted by the local Mobilenet network, and the last 6 frames are extracted by the cloud-based EfficientNet-B6 network.

[0052] (2) When there are fewer than 9 frames of video ultrasound area images, the first 3 frames are used to extract features from the local Mobilenet network, and the remaining frames are used to extract features from the cloud-based EfficientNet-B6 network.

[0053] (3) When there are fewer than 3 frames of video ultrasound area images, the Mobilenet network segments and extracts all images and uses them as the final output features of the system.

[0054] 4. Dual-network joint operation (inference)

[0055] The clinical ultrasound video to be identified is input frame by frame into the trained dual-network ultrasound nodule segmentation system. The specific operation sequence is the same as in step 3.

[0056] A concrete example is as follows: Each case is represented by an independent ultrasound video. Joint segmentation and extraction are performed on the video images: a nodule segmentation feature is output sequentially for every 9 frames; the first three frames of the 9 images are input into the MobileNet network (if there are fewer than three frames, all features are extracted by the MobileNet network), and the remaining frames are input into the EfficientNet-B6 network, with each neural network extracting features separately; finally, the features extracted by the MobileNet and EfficientNet-B6 networks are concatenated in the cloud to output a report including all nodule segmentation feature results.

[0057] It should be noted that the lightweight neural network and deep convolutional neural network used in the specific implementation of this invention can achieve the expected results with the same type of network, and the training parameters and methods used in this invention are the same as those in the prior art. Therefore, the network structure and pre-training process will not be described in detail here.

Claims

1. A method for ultrasonic nodule segmentation based on cloud computing and portable terminals, characterized in that, Includes the following steps: (1) Take a sufficient amount of ultrasound videos with pathological images of nodules, and after cropping and labeling, obtain a training dataset; (2) Deploy a lightweight convolutional neural network on a local portable terminal and a deep convolutional neural network on a cloud server to construct a dual-network ultrasound nodule segmentation system; In a dual-network nodal segmentation system, the loss function is defined as follows: ； in, Indicates the first The first image sequence One ultrasound image; Indicates and The corresponding mask output; This represents the predicted output of a lightweight convolutional neural network. This represents the network parameters of a lightweight convolutional neural network. This represents the predicted output of a deep convolutional neural network. This represents the network parameters of a deep convolutional neural network; Indicates batch video from 1 to , Indicates the number of videos in the batch; This indicates the number of images input to the lightweight convolutional neural network. Indicates the input image sequence from 1 to , Indicates the number of images in the input image sequence; (3) The two neural networks are jointly trained using the training dataset. The training process includes: a lightweight convolutional neural network first performs nodule segmentation and feature extraction on some frame images in the video, and then transmits the feature data and the remaining frame images to the cloud server via the network; a deep convolutional neural network performs nodule segmentation and feature extraction on the remaining frame images, and then splices the two extracted features. (4) Input the clinical ultrasound video to be identified frame by frame into the trained dual-network ultrasound nodule segmentation system to perform joint segmentation and extraction: The lightweight convolutional neural network first performs nodule segmentation and feature extraction on some frames of the video, and then transmits the feature data and the remaining frame images to the cloud server through the network; the deep convolutional neural network performs nodule segmentation and feature extraction on the remaining frame images, and then splices the two extracted features; finally, a report including all nodule segmentation feature results is output. Each case is represented by an independent ultrasound video. During joint training or joint segmentation and extraction, a nodule segmentation feature is jointly output for every 9 frames of ultrasound area images, with each ultrasound video serving as a unit. The first 3 frames are used to extract features from a local lightweight convolutional neural network, while the last 6 frames are used to extract features from a cloud-based deep convolutional neural network. When there are fewer than 9 frames of ultrasound area images in the video, the first 3 frames are used to extract features from a local lightweight convolutional neural network, while the remaining frames are used to extract features from a cloud-based deep convolutional neural network. When there are fewer than 3 frames of ultrasound area images in the video, the lightweight convolutional neural network segments and extracts features from all images, which is then used as the final output feature of the system.

2. The method according to claim 1, characterized in that, When creating the training dataset, the images are first split into frames, and the effective segments and ultrasound regions in the frame images are marked. Then, the original images are cropped according to the ultrasound region information, and nodules in the ultrasound region images within the effective segments are labeled according to clinical medical judgment criteria. In the training dataset, a mask binary image of the same size as the ultrasound region image of the nodule is generated for each nodule. Areas with non-zero pixels in the image indicate that a nodule exists at the corresponding position in the ultrasound region image, and vice versa.

3. The method according to claim 1, characterized in that, The local portable terminal refers to a portable ultrasound device capable of deploying a lightweight convolutional neural network and possessing computing and operational capabilities, or a portable computing device connected to the ultrasound device; the portable computing device refers to a tablet computer, smartphone, laptop computer, or PC host.

4. The method according to claim 1, characterized in that, The lightweight convolutional neural network is the Mobilenet network, whose structure includes: nine depthwise separable convolutions that form the backbone of the network, each of which can be decomposed into a depthwise convolution and a 1×1 convolution; and a convolutional layer with a kernel size of 3×3 and a stride of 2.

5. The method according to claim 1, characterized in that, The deep convolutional neural network is the EfficientNet-B6 network, whose structure includes: seven moving inverse bottleneck convolutional modules that form the network backbone, and a convolutional layer with a kernel size of 3×3 and a stride of 2; each moving inverse bottleneck convolutional module consists of three parts: two regular convolutions, one depthwise separable convolution, and one attention module.

Images