Medical image classification and annotation method, system and server

Abstract

The present invention discloses a classification and annotation method, system and server for medical images, and belongs to the field of medical image technology. The method comprises: obtaining an original medical image and sending it to a manual annotation terminal; obtaining a manual annotation map returned by the manual annotation terminal and obtaining the pixel boundary of the annotation area through an edge extraction algorithm; processing the original medical image through a pre-trained full convolutional neural network model to obtain a feature map with a size of the original medical image and a depth of the number of annotation types; comparing the feature map with the manual annotation map at the pixel boundary; if the results of the pixel point on the manual annotation map and the feature map are consistent, the pixel point is not corrected; otherwise, multiple pixel points around the pixel point on the feature map are selected, and the category with the largest number of multiple pixel points is used as the category of the pixel point; and the corrected feature map is output as the network annotation map.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a classification and annotation method and system, and a server for medical images. Background Art

[0002] Advanced medical imaging technologies such as ultrasound imaging and magnetic resonance imaging have produced a large number of two-dimensional and three-dimensional medical images for medical diagnosis. These images contain a variety of information such as pathological tissues and organs, which require medical staff to make judgments based on their professional knowledge. However, compared with conventional images, medical imaging has the characteristics of poor contrast and high noise, which causes problems such as time-consuming diagnosis and easy errors. By pre-segmenting and labeling medical images and quantitatively separating the target area from the background area, it can not only reduce the workload and cost of medical experts in the process of disease diagnosis, but also reduce the error rate of manual processes, thereby improving diagnostic efficiency and accuracy.

[0003] At present, the segmentation of medical images mostly adopts traditional image segmentation methods based on thresholds, edge detection, etc. These methods are easily affected by noise and it is difficult to ensure the continuity and closure of the segmentation edges. Artificial intelligence algorithms represented by neural networks have been maturely applied in image processing fields such as autonomous driving, and have become an emerging technology for solving medical image processing problems. Summary of the invention

[0004] In order to solve the above problems, the embodiments of the present invention provide a classification and annotation method and system and server for medical images. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present invention provides a classification and annotation method for medical images, the method comprising:

[0006] S101: Acquire the original medical image and send it to the manual annotation terminal;

[0007] S102: Acquire a manual annotation image returned by the manual annotation terminal and obtain pixel boundaries of the annotation area through an edge extraction algorithm, wherein the manual annotation image is obtained by manually classifying and annotating a target area on the original medical image by the manual annotation terminal according to a predetermined strategy;

[0008] S103: Processing the original medical image by a pre-trained fully convolutional neural network model to obtain a feature map with a size equal to the original medical image and a depth equal to the number of annotation types, wherein the layer number of the point with the largest pixel value in the depth direction of the feature map is the labeled category number;

[0009] S104: At the pixel boundary, the feature map is compared with the manual annotation map; if the results of the pixel point on the manual annotation map and the feature map are consistent, the pixel point is not corrected; otherwise, multiple pixels around the pixel point on the feature map are selected, and the category with the largest number of pixels among the multiple pixels is used as the category of the pixel point;

[0010] S105: Output the feature map processed in step S104 as a network annotation map.

[0011] Among them, in the manual labeling terminal, the edge contour of the target area is manually drawn, the area enclosed by the edge contour is filled by the connected graph algorithm, and the filled area is manually classified and labeled, and the manual labeling map corresponds to the category marked in the feature map.

[0012] Specifically, the labeling method of the manually labeled terminal in the embodiment of the present invention is:

[0013] S201: manually selecting a brush of a specific color and a thick line type, where different colors represent different tissues and correspond to corresponding annotation categories;

[0014] S202: manually use a brush to outline the general outline of the target area;

[0015] S203: Using a connected graph algorithm, marking the pixel points within the contour as a brush color to fill the area enclosed by the edge contour;

[0016] S204: manually select a thin line type to repair the edge of the area;

[0017] S205: If there are still areas to be labeled, repeat steps S201-S204. If not, obtain a manually labeled map.

[0018] Furthermore, the classification and annotation method provided by the present invention also includes: each original medical image has a unique data identification code; in step S101, the original medical image and the corresponding data identification code are obtained and sent together to the manual annotation terminal; in step S102, the manual annotation image and the corresponding data identification code are obtained, and in step S105, the network annotation image and the corresponding data identification code are output; the data identification code is generated by:

[0019] S301: The original medical image X is subjected to a linear projection matrix W matrix operation to obtain a vector Y;

[0020]

[0021] Among them, the size of the original medical image X is: width is w, height is 3h; the size of the linear projection matrix W is 1*w; the size of the vector Y is 1*3h; the linear projection matrix W is calculated according to the following formula:

[0022]

[0023] S302: round each element of vector Y down to the nearest integer, so that each element of vector Y is between [0, 255].

[0024] S303: Map the Y vector to a specific digital identification code ID through a hash function; if the specific digital identification code ID is the same as an existing image ID, add a suffix identifier to the specific digital identification code ID to distinguish it from the existing image ID; the specific digital identification code ID with or without the suffix identifier is the data identification code.

[0025] Wherein, step S103 specifically includes:

[0026] S401: Through multiple Convolution kernel and Max pooling extracts feature information of the original medical image and obtains the initial feature map;

[0027] S402: Downsample the initial feature map to a series of sizes through average pooling, Convolution kernel convolution processing; then restore the series size feature map to the size of the initial feature map through bilinear interpolation method and equal-size convolution in turn to obtain the restored feature map;

[0028] S403: splicing the restored feature map and the initial feature map to obtain a spliced ​​feature map;

[0029] S404: The concatenated feature map is restored to the original medical image size by bilinear interpolation method and equal-size convolution in turn, and then a softmax algorithm is used to obtain a feature map with a size of the original medical image and a depth of the number of annotation types.

[0030] Preferably, the classification and labeling method provided by the present invention also includes: selecting the original medical image with a small error between the feature map and the manually labeled map as a high-quality sample, using the high-quality sample as the input of the full convolutional neural network model, and using the network labeling map corresponding to the original medical image as the output of the full convolutional neural network model, and optimizing the training of the full convolutional neural network model.

[0031] Specifically, in step S104, if the proportion of pixel points at the pixel boundary whose results are inconsistent between the feature map and the manually labeled map is less than a predetermined threshold, it is considered that the error between the feature map and the manually labeled map is small, and the corresponding original medical image is a high-quality sample; the optimization training method includes: using the original labeled medical image as a verification set, and performing k-fold cross-validation on the high-quality samples.

[0032] Specifically, in step S104: the number of the plurality of pixel points is 15-25.

[0033] On the other hand, an embodiment of the present invention further provides a classification and annotation system for medical images, the system comprising:

[0034] User: used to upload original medical images to the server and display network annotation images;

[0035] Central database: used to obtain and store the original medical images uploaded by users, distribute the original medical images to the server, store the network annotation map and send the network annotation map to users;

[0036] Server: used to distribute the original medical image to the manual annotation terminal, process the original medical image through the pre-trained full convolutional neural network model to obtain a feature map with the size of the original image and the depth of the number of annotation types, obtain the manual annotation map and process the manual annotation map through the edge extraction algorithm to obtain the pixel boundary of the annotation area; at the pixel boundary, modify the feature map by comparing it with the manual annotation map to obtain the network annotation map, and upload the network annotation map to the central database;

[0037] Manual annotation terminal: used to display the original medical image and the manually annotated image, manually annotate the original medical image to obtain the manually annotated image, and upload the manually annotated image to the server.

[0038] In another aspect, an embodiment of the present invention further provides a server, comprising:

[0039] Data transceiver module: used to receive the original medical images sent by the central database, send the original medical images to the manual annotation terminal, and send the network annotation map to the central database;

[0040] Image processing module: used to process the manually labeled image through edge extraction algorithm to obtain the pixel boundary of the labeled area;

[0041] Fully convolutional neural network module: used to process the original medical image through a pre-trained fully convolutional neural network model to obtain a feature map with a size of the original medical image and a depth of the number of annotation types, and the layer number of the point with the largest pixel value in the depth direction of the feature map is the labeled category number;

[0042] Correction module: used to compare the feature map with the manual annotation map at the pixel boundary; if the results of the pixel point on the manual annotation map and the feature map are consistent, the pixel point will not be corrected; otherwise, multiple pixels around the pixel point on the feature map are selected, and the category with the largest number of pixels among the multiple pixels is used as the category of the pixel point; the corrected feature map is used as the network annotation map.

[0043] Among them, the fully convolutional neural network module provided by the embodiment of the present invention includes:

[0044] Feature map extraction unit: used to extract feature information of the original medical image through multiple convolutions and pooling to obtain the initial feature map;

[0045] Multi-scale convolutional pooling unit: used to downsample the initial feature map to a series of sizes through average pooling, The convolution kernel is used for convolution processing, and then the series size feature maps are restored to the size of the initial feature map through bilinear interpolation method and equal-size convolution in turn to obtain the restored feature map;

[0046] Splicing unit: used for splicing the restored feature map and the initial feature map to obtain a spliced ​​feature map;

[0047] Upsampling unit: used to restore the spliced ​​feature map to the original medical image size through bilinear interpolation method and equal-size convolution, and then obtain a feature map with the size of the original medical image and the depth of the number of annotation types through the softmax algorithm.

[0048] In another aspect, an embodiment of the present invention further provides a manual annotation terminal, including: an original image display module: used to display the original medical image and enable manual annotation. During manual annotation, a rough outline of a target area is firstly drawn using a thick line type, and then the edge of the area is repaired using a thin line type;

[0049] Control module: used for processing the annotated original medical image to obtain a manually annotated image;

[0050] Manually annotated graph display module: used to display manually annotated graphs;

[0051] Selection module: used to select the brush color and line thickness. Different colors represent different organizations and correspond to corresponding annotation categories.

[0052] Communication module: used to receive original medical images and upload manually annotated images.

[0053] The beneficial effects of the technical solution provided by the embodiment of the present invention are as follows: the present invention constructs a medical image segmentation and annotation method and system based on a fully convolutional neural network, which can realize an end-to-end mapping relationship between pixels of original medical images and pixels of network annotated images, and the tensor operation greatly improves the image processing speed; compared with traditional image processing algorithms, the neural network annotation method used in the present invention does not require artificially designed image processing operators, and can be used for annotation of medical images of different types and sizes when using excellent training data sets; by integrating the annotation results output by the convolutional neural network and the manual annotation results, the neural network weights are optimized, thereby improving the results of image segmentation and annotation, realizing positive feedback in the image processing process, and continuously improving the efficiency and accuracy of medical image diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a classification and annotation method for medical images provided by an embodiment of the present invention;

[0055] Figure 2 is a flow chart of a method for manually labeling a terminal provided by an embodiment of the present invention;

[0056] Figure 3 This is an operation interface diagram of a manual annotation terminal provided by an embodiment of the present invention;

[0057] Figure 4 is an operation flow chart of a manual annotation terminal provided by an embodiment of the present invention;

[0058] Figure 5 is a flow chart of a method for generating a data identification code provided by an embodiment of the present invention;

[0059] Figure 6 is a specific flow chart of step S103;

[0060] Figure 7 It is the specific processing method of step S103;

[0061] Figure 8 is a principle block diagram of a medical image classification and annotation system provided by an embodiment of the present invention;

[0062] Fig. 9 is a principle block diagram of a server provided by an embodiment of the present invention;

[0063] Fig.10 It is a principle block diagram of a fully convolutional neural network module provided by an embodiment of the present invention;

[0064] Fig.11 It is a principle block diagram of a manual annotation terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0066] Example 1

[0067] See also Figure 1 Embodiment 1 discloses a classification and annotation method for medical images, the method comprising:

[0068] S101: Acquire the original medical image and send it to the manual annotation terminal, and also obtain the data identification code.

[0069] S102: Obtain the manual annotation image (and the corresponding data identification code) returned by the manual annotation terminal and obtain the pixel boundary of the annotation area through the edge extraction algorithm; wherein the manual annotation image is obtained by the manual annotation terminal by manually classifying and annotating the target area on the original medical image according to a predetermined strategy; in this embodiment, only the outline of the target area needs to be annotated (i.e., rough annotation), the work intensity is low, and the annotation speed is very fast. Among them, the pixel boundary is the area near which the annotation result can be wrong.

[0070] S103: Process the original medical image through the pre-trained fully convolutional neural network model to obtain a feature map with a size of the original medical image and a depth of the number of annotation types. The layer number of the point with the largest pixel value in the depth direction of the feature map (corresponding to Figure 7 The serial number (N) in the table is the marked category number.

[0071] S104: At the pixel boundary, compare the feature map with the manual annotation map (pixel point). If the results of the pixel point on the manual annotation map and the feature map are consistent, the pixel point is not corrected. Otherwise, multiple (such as 20) pixel points around the pixel point on the feature map are selected, and the category with the largest number of multiple pixel points is used as the category of the pixel point to correct the corresponding pixel point in the feature map. This step realizes the correction of the feature map, which can not only improve the accuracy of the network annotation map, but also be used for feedback optimization training. Specifically, among the 20 pixel points around the differently annotated pixel points, the category number of 8 pixel points is category 1, the category number of 5 pixel points is category 2, the category number of 3 pixel points is category 3, the category number of 2 pixel points is category 4, and the category number of 2 pixel points is category 4, then the category with the largest number is category 1, and the category number of the differently annotated pixel points is category 1. Specifically, in step S104: the number of multiple pixel points is 15-25, specifically 20; the number of pixel points is selected according to the complexity of the image, and the more complex the image, the larger the number.

[0072] S105: The corrected feature map (after processing in step S104) (and the corresponding data identification code) is output as the network annotation map; of course, the manual annotation map can also be output together.

[0073] Among them, in the manual labeling terminal, the edge contour of the target area is manually drawn, and the area enclosed by the edge contour is filled through the connected graph algorithm (specifically, the same color as the edge contour can be used), and the filled area is manually classified and labeled (the color can be directly used for classification, or it can be defined after filling). The categories of the manually labeled image correspond to those of the feature map annotations. Manual labeling should be carried out according to certain strategies (such as defining the color of cancer tissue, the color of diseased tissue, the color of normal tissue, etc. according to actual conditions) to ensure that the categories of the manually labeled image correspond to those of the feature map annotations.

[0074] Specifically, see Figure 2-4 , the labeling method of the manually labeled terminal in the embodiment of the present invention is:

[0075] S201: Manually select a brush of a specific color and a thick line type, where different colors represent different tissues and correspond to corresponding annotation categories.

[0076] S202: Manually use a brush to outline the general outline of the target area (such as using a thick dotted line for outlining), and click to confirm.

[0077] S203: Using a connected graph algorithm, the pixel points within the contour are marked as a brush color to fill the area enclosed by the edge contour.

[0078] S204: Manually select a thin line type and repair the edge of the area to make the marked area as correct as possible.

[0079] S205: If there are still areas to be marked, repeat steps S201-S204 (using corresponding colors for different parts). If not, a manually marked image is obtained, then click Next to mark the next image.

[0080] S206: If there is an image that needs to be modified, click the previous image to review it.

[0081] S207: After all images are labeled, they are packaged and sent to the server.

[0082] The marking process of the present invention can be implemented on a conventional PC, preferably on a touch screen device to facilitate outlining.

[0083] Specifically, the operator logs in to the account assigned by the server to obtain the original medical image. The manual annotation terminal has the functions of displaying, switching, and storing medical images. After completing the annotation of a medical image, it is temporarily stored in the manual annotation terminal, and the next image is annotated. After completing the rough annotation of a group of, for example, 20 images, the manual annotation terminal returns the annotation data to the server, which then processes it further. Similarly, the manual annotation terminal can also work online. For example, each time the user completes the rough annotation of a medical image, the terminal returns the roughly annotated image to the server and downloads a new medical image from the server for annotation.

[0084] Furthermore, the classification labeling method provided by the present invention also includes: each original medical image has a unique data identification code; in step S101, the original medical image and the corresponding data identification code are obtained and sent together to the manual labeling terminal; in step S102, the manual labeling image and the corresponding data identification code are obtained, and in step S105, the network labeling image and the corresponding data identification code are output.

[0085] Specifically, after the central database generates the data identification code, it sends the original medical image and the data identification code to the server, and the server randomly distributes the medical image to be annotated (original medical image) to the manual annotation terminal. After the manual annotation terminal receives the data, the operator manually annotates it roughly and sends the roughly annotated manual annotation image back to the server. The server performs computational processing on the roughly annotated manual annotation image to extract the accurate pixel edges of the roughly annotated features, and then combines the fully convolutional neural network model to achieve pixel-level segmentation and multi-classification of the target tissue or organ. After the image annotation is completed, each server sends the annotated image (network annotation image) back to the central database, which then returns it to the user. The central database can generate a unique data identification code for each original medical image uploaded by the user to establish storage and indexing, and the data identification code is transmitted to the server together with the corresponding image. After the server completes the processing of the medical image, it sends the image back to the central database, and the central database establishes a one-to-one correspondence between the annotated medical image and the original medical image according to the data identification code and stores it. Users can extract original medical images, manually annotated images, and network annotation images from the central database as needed.

[0086] Specifically, see Figure 5 , the method for generating the data identification code in the embodiment of the present invention is:

[0087] S301: The original medical image X is subjected to a linear projection matrix W matrix operation to obtain a vector Y;

[0088]

[0089] Among them, the size of the original medical image X is: width is w, height is 3h; the size of the linear projection matrix W is 1*w; the size of the vector Y is 1*3h; the linear projection matrix W is calculated according to the following formula:

[0090] .

[0091] S302: Each element of vector Y is rounded down to the nearest integer, so that each element of vector Y is between [0, 255].

[0092] S303: Map the Y vector to a specific digital identification code ID through a hash function; if the specific digital identification code ID is the same as an existing image ID, add a suffix identifier to the specific digital identification code ID to distinguish it from the existing image ID; the specific digital identification code ID with (the same as the existing image ID) or without (not the same as the existing image ID) the suffix identifier is the data identification code.

[0093] Among them, see Figure 6 and 7 , step S103 specifically includes:

[0094] S401: Extract feature information of the original medical image (specifically 512*512*3) through multiple convolutions and pooling, while reducing the image size and deepening the depth of the feature map, thereby extracting the deep feature information contained in the original image and obtaining an initial feature map.

[0095] S402: Downsample the initial feature map to a series of sizes (such as 8*8*256, 4*4*256, 2*2*256 and 1*1*256) through average pooling. Convolution kernel convolution processing (the results are 8*8*256, 4*4*256, 2*2*256 and 1*1*256 respectively). Then, the series size feature maps are restored to the initial feature map size through bilinear interpolation method and equal size convolution to obtain the restored feature map.

[0096] S403: Concatenate the restored feature map and the initial feature map to obtain a concatenated feature map, which is consistent with conventional techniques.

[0097] S404: The concatenated feature map is restored to the original medical image size by bilinear interpolation method and equal-size convolution in turn, and then a softmax algorithm is used to obtain a feature map with a size of the original medical image and a depth of the number of annotation types (512*512*N, where N is the number of annotation types).

[0098] Specifically, step S401 includes: Convolution kernel and Max pooling extracts the feature information of the original medical image and obtains the initial feature map. Specifically, it is convolution 3*3*64, max pooling 2*2, convolution 3*3*256, max pooling 2*2, and convolution 3*3*512.

[0099] Preferably, the classification and labeling method provided by the present invention also includes: selecting the original medical image with a smaller error between the feature map and the manually labeled map as a high-quality sample, using the high-quality sample as the input of the full convolutional neural network model, using the network labeling map corresponding to the original medical image as the output of the full convolutional neural network model, and optimizing the training of the full convolutional neural network model. The optimization training is a conventional technology in the field, and a detailed description is omitted in this embodiment.

[0100] Specifically, in step S104, if the proportion of pixels whose results in the feature map and the manually labeled map are inconsistent in the pixel boundary (pixel set) (the proportion of the number of pixels) is less than a predetermined threshold (designed according to actual needs, specifically 1%), it is considered that the error between the feature map and the manually labeled map is small, and the corresponding original medical image is a high-quality sample. The optimization training method includes: using the original labeled medical image (stored in the central database for training) as a validation set, performing k-fold cross validation on high-quality samples, and improving the average intersection-over-union ratio between the output of the full convolutional neural network model and the actual labeled map of the image.

[0101] Example 2

[0102] See also Figure 8 Embodiment 2 discloses a classification and annotation system for medical images, the system comprising:

[0103] User: Used to upload original medical images to the server and display network annotation images.

[0104] Central database: used to obtain and store original medical images uploaded by users, distribute original medical images to servers, store network annotation maps and send network annotation maps to users.

[0105] Server: used to distribute the original medical image to the manual annotation terminal, process the original medical image through a pre-trained fully convolutional neural network model to obtain a feature map with a size of the original image and a depth of the number of annotation types, obtain the manual annotation map and process the manual annotation map through an edge extraction algorithm to obtain the pixel boundary of the annotation area; at the pixel boundary, correct the feature map by comparing it with the manual annotation map (see step S104 for details) to obtain a network annotation map, and upload the network annotation map to the central database.

[0106] Manual annotation terminal: used to display the original medical image and the manually annotated image, manually annotate the original medical image to obtain the manually annotated image, and upload the manually annotated image to the server.

[0107] Among them, users such as hospitals, medical research institutes, and medical companies upload medical images to the central database after acquiring them.

[0108] Among them, the central database can be connected to one or more servers according to the actual number of medical images, and automatically distribute the original medical images to different servers for processing. When the number of medical images to be processed is small, the server and the central database can be merged into one. After receiving the original medical image, the server automatically distributes the image to one or more manual annotation terminals for rough annotation. The central database can be set up in the user's unit local area network, cloud storage, etc., and users can upload data through wireless networks, Ethernet, USB and other communication methods. The central database sets a dedicated account and password for each user, and assigns read, write, change and other permissions to ensure data security. After obtaining the medical images uploaded by the user, the central database generates a unique digital identification code for each image, binds it to the corresponding user, and distributes the original medical image and digital identification code to the server.

[0109] The server can adopt a variety of computing platforms such as physical servers, cloud servers, personal computers, etc. The server assigns an account and password to each manual annotation terminal, and after receiving the original medical image and digital identification code, distributes it to the manual annotation terminal, and the manual annotation unit performs rough annotation.

[0110] The manual annotation terminal can run on multiple operating systems such as Windows, Android, and IOS, and can synchronize data across platforms. When the operator completes the rough annotation of the original image, the manually annotated image can be sent back to the server.

[0111] The manual annotation terminal transmits the manually annotated map back to the server; the server transmits the processed network annotation map back to the central database for users to download. Figure 8 K, M, N, etc. refer to unknown quantities. In practical applications, the number of servers can be increased or decreased according to the actual number of images, and the same instructions and full convolutional neural network model are run on each server. Here, the central database and the server refer to the corresponding functional units. In practical applications, the central database and the server can also be integrated into one. The manual annotation terminal can be a computer, a laptop, a tablet computer, a mobile phone, etc., and can run on operating systems such as Windows, Android, and IOS. The number can be increased or decreased according to the actual application.

[0112] Example 3

[0113] See also Fig. 9 , Embodiment 3 discloses a server, including:

[0114] Data transceiver module: used to receive and store original medical images sent by the central database, send original medical images to the manual annotation terminal, and send network annotation images to the central database. It is a common structure.

[0115] Image processing module: used to process the manually labeled image through edge extraction algorithm to obtain the pixel boundary of the labeled area.

[0116] Fully convolutional neural network module: used to process the original medical image through a pre-trained fully convolutional neural network model to obtain a feature map with a size of the original medical image and a depth equal to the number of annotation types. The layer number of the point with the largest pixel value in the depth direction of the feature map is the labeled category number.

[0117] Correction module: used to compare the feature map with the manual annotation map at the pixel boundary; if the results of the pixel point on the manual annotation map and the feature map are consistent, the pixel point will not be corrected; otherwise, multiple pixels around the pixel point on the feature map are selected, and the category with the largest number of pixels among the multiple pixels is used as the category of the pixel point to correct the corresponding pixel point in the feature map; the corrected feature map (after processing in step S104) is used as the network annotation map.

[0118] Storage module: used to store data.

[0119] Furthermore, the server also includes a feedback module for selecting original medical images with smaller errors between feature maps and manually annotated maps as high-quality samples, using the high-quality samples as input of the full convolutional neural network model, and using the network annotation map corresponding to the original medical image as output of the full convolutional neural network model to optimize the training of the full convolutional neural network model.

[0120] Among them, see Fig.10 , the fully convolutional neural network module provided by the embodiment of the present invention includes:

[0121] Feature map extraction unit: used to extract the feature information of the original medical image through multiple convolutions and pooling to obtain the initial feature map; specifically, through multiple Convolution kernel and Max pooling extracts the feature information of the original medical image and obtains the initial feature map.

[0122] Multi-scale convolutional pooling unit: used to downsample the initial feature map to a series of sizes through average pooling, After convolution kernel convolution processing, the series size feature maps are restored to the initial feature map size through bilinear interpolation method and equal-size convolution to obtain the restored feature map.

[0123] Splicing unit: used for splicing the restored feature map and the initial feature map to obtain a spliced ​​feature map;

[0124] Upsampling unit: used to restore the spliced ​​feature map to the original medical image size through bilinear interpolation method and equal-size convolution, and then obtain a feature map with the size of the original medical image and the depth of the number of annotation types through the softmax algorithm.

[0125] Example 4

[0126] See also Fig.11 Embodiment 4 discloses a manual annotation terminal, including: an original image display module: used to display the original medical image and enable manual annotation. During manual annotation, a rough outline of the target area is firstly drawn with a thick line type, and then the edge of the area is repaired with a thin line type.

[0127] Control module: used to process the annotated original medical image to obtain the manually annotated image, and control the entire manual annotation terminal, such as outputting display data to the original image display module and the manually annotated image display module.

[0128] Manually annotated graph display module: used to display manually annotated graphs.

[0129] Selection module: used to select the pen color and line thickness. Different colors represent different organizations and correspond to corresponding annotation categories. Specifically, multiple line types (at least two) and multiple colors (defined according to a predetermined strategy) can be selected.

[0130] Communication module: used to receive original medical images and upload manually annotated images.

[0131] Specifically, the original image display module, the manually annotated image display module and the selection module can be implemented by the same touch display screen.

[0132] Specifically, the original image display module generates a blank canvas of the same size as the original image. The selection module can select different colored brushes and different thick and thin line types. It can also include a confirmation button, a next button, a previous button and a completion button. After the operator completes the rough annotation of the original image, click the confirmation button, the annotation is saved to the annotation image cache area and the filled annotation image is displayed on the manual annotation image display module; the operator clicks the next button to jump to the next unannotated original image for annotation; the previous button can review the previously annotated image. If the image is modified during the review, the modification result will be saved to the annotation image cache area. When the original image display area is blank, it means that the server has not currently assigned a task, then click the completion button to complete the annotation.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A classification and labeling method for medical images, characterized in that: The method comprises: S101: Acquire the original medical image and send it to the manual annotation terminal; S102: Acquire a manual annotation image returned by the manual annotation terminal and obtain pixel boundaries of the annotation area through an edge extraction algorithm, wherein the manual annotation image is obtained by manually classifying and annotating a target area on the original medical image by the manual annotation terminal according to a predetermined strategy; S103: Processing the original medical image by a pre-trained fully convolutional neural network model to obtain a feature map with a size equal to the original medical image and a depth equal to the number of annotation types, wherein the layer number of the point with the largest pixel value in the depth direction of the feature map is the labeled category number; S104: At the pixel boundary, the feature map is compared with the manual annotation map; if the results of the pixel point on the manual annotation map and the feature map are consistent, the pixel point is not corrected; otherwise, multiple pixels around the pixel point on the feature map are selected, and the category with the largest number of pixels among the multiple pixels is used as the category of the pixel point; S105: Outputting the feature map processed in step S104 as a network annotation map; At the manual labeling terminal, the edge contour of the target area is manually drawn, the area enclosed by the edge contour is filled by the connected graph algorithm, and the filled area is manually classified and labeled, and the manual labeling map corresponds to the category marked in the feature map; The original medical image with a small error between the feature map and the manually labeled map is selected as a high-quality sample, and the high-quality sample is used as the input of the full convolutional neural network model. The network labeled map corresponding to the original medical image is used as the output of the full convolutional neural network model, and the full convolutional neural network model is optimized and trained; if the proportion of pixels with inconsistent results between the feature map and the manually labeled map at the pixel boundary is less than a predetermined threshold, it is considered that the error between the feature map and the manually labeled map is small; Step S103 specifically includes: S401: Through multiple Convolution kernel and Max pooling extracts feature information of the original medical image and obtains the initial feature map; S402: Downsample the initial feature map to a series of sizes through average pooling, Convolution kernel convolution processing; then restore the series size feature map to the size of the initial feature map through bilinear interpolation method and equal-size convolution in turn to obtain the restored feature map; S403: splicing the restored feature map and the initial feature map to obtain a spliced ​​feature map; S404: The concatenated feature map is restored to the original medical image size by bilinear interpolation method and equal-size convolution in turn, and then a softmax algorithm is used to obtain a feature map with a size of the original medical image and a depth of the number of annotation types.

2. The classification and labeling method for medical images according to claim 1, characterized in that: The labeling method of the manually labeled terminal is: S201: manually selecting a brush of a specific color and a thick line type, where different colors represent different tissues and correspond to corresponding annotation categories; S202: manually use a brush to outline the general outline of the target area; S203: Using a connected graph algorithm, marking the pixel points within the contour as a brush color to fill the area enclosed by the edge contour; S204: manually select a thin line type to repair the edge of the area; S205: If there are still areas to be labeled, repeat steps S201-S204. If not, obtain a manually labeled map.

3. The classification and labeling method for medical images according to claim 1, characterized in that: The method further includes: each original medical image has a unique data identification code; in step S101, the original medical image and the corresponding data identification code are obtained and sent together to the manual annotation terminal; in step S102, the manual annotation image and the corresponding data identification code are obtained, and in step S105, the network annotation image and the corresponding data identification code are output; The method for generating the data identification code is: S301: The original medical image X is subjected to a linear projection matrix W matrix operation to obtain a vector Y; Among them, the size of the original medical image X is: width is w, height is 3h; the size of the linear projection matrix W is 1*w; the size of the vector Y is 1*3h; the linear projection matrix W is calculated according to the following formula: S302: round each element of vector Y down to the nearest integer, so that each element of vector Y is between [0, 255]. S303: Map the Y vector to a specific digital identification code ID through a hash function; if the specific digital identification code ID is the same as an existing image ID, add a suffix identifier to the specific digital identification code ID to distinguish it from the existing image ID; the specific digital identification code ID with or without the suffix identifier is the data identification code.

4. The classification and labeling method for medical images according to claim 1, characterized in that: In step S104, the optimization training method includes: using the original annotated medical images as a validation set, and performing k-fold cross validation on high-quality samples.

5. A server, characterized in that: include: Data transceiver module: used to receive the original medical images sent by the central database, send the original medical images to the manual annotation terminal, and send the network annotation map to the central database; Image processing module: used to process the manually labeled image through edge extraction algorithm to obtain the pixel boundary of the labeled area; Fully convolutional neural network module: used to process the original medical image through a pre-trained fully convolutional neural network model to obtain a feature map with a size of the original medical image and a depth of the number of annotation types, and the layer number of the point with the largest pixel value in the depth direction of the feature map is the labeled category number; Correction module: used to compare the feature map with the manually labeled map at the pixel boundary; If the pixel point on the manual annotation map and the feature map has the same result, the pixel point will not be corrected; otherwise, multiple pixels around the pixel point on the feature map will be selected, and the category with the largest number of pixels among the multiple pixels will be used as the category of the pixel point; Use the corrected feature map as the network annotation map; The fully convolutional neural network module includes: Feature map extraction unit: used to extract feature information of the original medical image through multiple convolutions and pooling to obtain the initial feature map; Multi-scale convolutional pooling unit: used to downsample the initial feature map to a series of sizes through average pooling, The convolution kernel is used for convolution processing, and then the series size feature maps are restored to the size of the initial feature map through bilinear interpolation method and equal-size convolution in turn to obtain the restored feature map; Splicing unit: used for splicing the restored feature map and the initial feature map to obtain a spliced ​​feature map; Upsampling unit: used to restore the spliced ​​feature map to the original medical image size through bilinear interpolation method and equal-size convolution, and then obtain a feature map with the size of the original medical image and the depth of the number of annotation types through the softmax algorithm; The server is also used to select original medical images with small errors between feature maps and manually labeled maps as high-quality samples, use the high-quality samples as input of the full convolutional neural network model, and use the network labeled map corresponding to the original medical image as output of the full convolutional neural network model to optimize the training of the full convolutional neural network model; wherein, if the proportion of pixels whose results are inconsistent between the feature map and the manually labeled map at the pixel boundary is less than a predetermined threshold, it is considered that the error between the feature map and the manually labeled map is small.

Images