A multi-level processing method and system for medical data

Abstract

The present invention provides a multi-level processing method and system for medical data, including converting a medical image into a pyramid image, dividing each layer of the pyramid image into grids; recording the viewing trajectory of a doctor to determine the concerned grids; performing edge detection on each layer of the medical pyramid image to determine the organ contour; annotating the organ type and the lesion location; labeling the organ contour corresponding to the concerned grids of each layer of the medical pyramid image as a detection target; performing network training; inputting the image to be viewed into the trained neural network model, outputting the position of the detection target, and displaying the detection result in a tree form. Through the above solution, the technical problem that doctors need to continuously zoom in and out when viewing high-resolution doctor images is solved.

Description

Technical Field

[0001] The present invention relates to the field of medical data processing, and in particular to a multi-level processing method and system for medical data. Background Art

[0002] Currently, the resolution of medical images is getting higher and higher. Due to the size of the display screen, a complete image detail usually cannot be displayed on one interface. When doctors view medical images, sometimes they need to view them macroscopically, such as viewing the blood vessel trend, etc., and sometimes they need to view very small details, such as the edge of the lesion, etc. Therefore, when doctors view medical images, they often need to perform zoom-in and zoom-out operations to find the target area to be viewed, resulting in doctors spending too much time on zooming and searching. Summary of the Invention

[0003] In order to solve the problems in the above background art, the present invention provides a multi-level processing method and system for medical data.

[0004] In one aspect of the present invention, a multi-level processing method for medical data is provided, including: converting a medical image into a pyramid image, the pyramid image being hierarchically composed of multiple medical images with different resolutions, and the resolution being higher at the lower layers; dividing each layer of the pyramid image into grids, and the number of grids being more at the lower layers; when a doctor views a medical image, recording the viewing trajectory of the doctor, and if not zooming in for viewing the next layer, determining the grid corresponding to the current image as the concerned grid; performing edge detection on each layer of the medical pyramid image to determine the organ contour; using artificial or machine learning methods to label the organ type and lesion location of the medical image; labeling the organ contour corresponding to the concerned grid of each layer of the medical pyramid image as the detection target to obtain a training sample; inputting the training sample into a neural network for training to obtain a neural network model capable of labeling the detection target; labeling the lesion location and organ type of the image to be viewed, converting the labeled image into a pyramid image and performing edge detection, and inputting the result image of the edge detection into the trained neural network model to output the location of the detection target; determining the corresponding target grid according to the output location of the detection target, and organizing the images corresponding to the target grid into a tree according to the pyramid image hierarchy relationship, and each node of the tree being displayed as a thumbnail.

[0005] Further, the pyramid image is divided into five layers, and the number of grids in each layer is 1, 4, 16, 64, and 256 respectively.

[0006] Further, when the time that the doctor stays on a certain image is greater than the first threshold, it is considered that the grid corresponding to the image is the concerned grid.

[0007] Further, select any tree node and display the selected node image in detail on the detailed browsing interface.

[0008] Further, when the target to be viewed is not found in the tree structure, manually search for the target position by zooming in and out of the detailed browsing interface. When the target position is found, manually add the target position to the tree structure.

[0009] On the other hand, the present invention also provides a multi-level processing system for medical data, which is characterized by including the following modules: a layering module for converting a medical image into a pyramid image, the pyramid image being hierarchically composed of a plurality of medical images with different resolutions, and the resolution being higher for the lower layers; a partitioning module for partitioning each layer of the pyramid image into grids, and the number of grids being larger for the lower layers; a first determination module for recording the viewing trajectory of a doctor when the doctor views a medical image, and if no further zooming is performed to view the next layer, determining the grid corresponding to the current image as the concerned grid; a second determination module for performing edge detection on each layer of the medical pyramid image to determine the organ contour; using artificial or machine learning methods to label the organ type and lesion location of the medical image; a labeling module for labeling the organ contour corresponding to the concerned grid of each layer of the medical pyramid image as the detection target to obtain a training sample; a training module for inputting the training sample into a neural network for training to obtain a neural network model capable of labeling the detection target; a detection module for labeling the lesion location and organ type of the image to be viewed, converting the labeled image into a pyramid image and performing edge detection, and inputting the result image of the edge detection into the trained neural network model; an output module for determining the corresponding target grid according to the output position of the detection target, organizing the image corresponding to the target grid into a tree according to the pyramid image level relationship, and displaying each node of the tree as a thumbnail.

[0010] Further, the pyramid image is divided into five layers, and the number of grids in each layer is 1, 4, 16, 64, and 256 respectively.

[0011] Further, it is considered that the grid corresponding to the image is the concerned grid only when the time the doctor stays on a certain image is greater than the first threshold.

[0012] Further, select any tree node and display the selected node image in detail on the detailed browsing interface.

[0013] Further, when the target to be viewed is not found in the tree structure, manually search for the target position by zooming in and out of the detailed browsing interface. When the target position is found, manually add the target position to the tree structure.

[0014] Through the above technical solutions, the present invention processes hospital images in layers, divides each layer into grids, collects the target samples that doctors want to see through the daily operations of doctors, and conducts neural network training to obtain a model that can detect the targets that doctors want to view, and displays the targets in a tree structure to facilitate the operations of doctors and reduce the workload of doctors frequently zooming in and out of images to search for targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0016] Figure 1 Schematic diagram of layer-by-layer processing of medical images;

[0017] Figure 2 Schematic diagram of the operation trajectory of doctors;

[0018] Figure 3 Schematic diagram of the target detection contour;

[0019] Figure 4 Schematic diagram of tree structure display. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Next, with reference to the drawings and specific embodiments, the present invention will be preferably described.

[0021] This embodiment solves the above problems through the following steps:

[0022] In one embodiment, the present invention provides a multi-level processing method for medical data, including the following steps:

[0023] Step S1, convert the medical image into a pyramid image, which is composed of multiple medical images with different resolutions in a hierarchical manner, and the resolution of the lower layer is higher.

[0024] The medical images described in the present invention can be any images generated during the medical process, such as CT images, MR images, etc., and can be two-dimensional images or three-dimensional images. In order to be able to display the lesion characteristics in detail, the resolution of current medical images is getting higher and higher, and the display cannot show all the details within one screen. In order to view the lesion information at different scales, doctors need to continuously perform zoom-in and zoom-out operations to search for the target location.

[0025] Since the resolution of the image is too high, on the one hand, it consumes a lot of resources during zooming and is prone to lag problems. On the other hand, it is inconvenient to record the viewing trajectory of the doctor. Therefore, the present invention further processes the medical image and converts the medical image into a pyramid image. The pyramid conversion algorithm belongs to a mature algorithm in the art, and the present invention will not give a detailed conversion description. As Figure 1 shown, it is a schematic diagram of the converted pyramid image. The bottom layer is the image with the highest resolution. The higher up, the lower the resolution, but the larger the range that can be displayed on the same monitor, making it easier to view macroscopic information.

[0026] Step S2, divide each layer of the pyramid image into grids, and the number of grids in the lower layer is more;

[0027] Furthermore, for the convenience of image calling, the present invention further divides each layer of the pyramid image into grids. As Figure 1 shown, each layer is divided into a plurality of grids. Since the resolution of the bottom layer is larger, it is divided into more grids; furthermore, each upper-layer grid corresponds to a plurality of lower-layer grids. As Figure 1 shown, the resolution of the first layer is the lowest and it has only one grid. The second layer is divided into four grids. Therefore, the first layer corresponds to four grids in the second layer; similarly, one grid in the second layer corresponds to four grids in the third layer.

[0028] Preferably, the pyramid image is divided into five layers, and the number of grids in each layer is 1, 4, 16, 64, and 256 respectively.

[0029] Step S3, when the doctor views the medical image, record the viewing trajectory of the doctor. If no further zooming is performed to view the next layer, determine the grid corresponding to the current image as the focus grid;

[0030] In order to record the historical operations of the doctor, record the operation trajectory of the doctor when the doctor views the medical image; as Figure 2 shown is the operation trajectory of the doctor, where the arrow is the zoom operation direction of the doctor; the doctor first performs a zoom operation for the first time to view the image in the lower left of the second layer, then continues to zoom and views the images in the third layer and the fourth layer in turn. After that, the doctor zooms out and views two images in the third layer. Then, the grids of one image in the fourth layer and two images in the third layer are the focus grids. Save the pyramid image and the corresponding focus grids for subsequent operations.

[0031] Preferably, in order to reduce errors, when the doctor stays on a certain image for more than the first threshold, it is considered that the grid corresponding to the image is the focus grid.

[0032] Step S4: Perform edge detection on each layer of the medical pyramid image to determine the organ contour; use artificial or machine learning methods to label the organ type and the lesion location on the medical image.

[0033] During the medical process, lesions usually occur within organs. When determining the surgical plan, doctors not only need to consider the lesion site, but also need to consider the associated organ sites, blood vessels, etc. Therefore, when viewing medical images, doctors usually zoom in to view the lesions and the associated organ sites. Organs usually have clear contours, and the organ contour can be determined by currently popular edge detection algorithms. The lesion contour is usually irregular and the boundary is not clear. In the present invention, the organ type and the lesion location are preferably labeled manually. At present, there are already some publicly available methods for determining the lesion location through machine learning. Further, the present invention can also use machine learning methods for labeling to determine the organ type and the lesion location.

[0034] Step S5: Label the organ contour corresponding to the attention grid of each layer of the medical pyramid image as the detection target to obtain training samples.

[0035] For the same human organ, the important nodes are usually similar. Therefore, when there is a lesion nearby, doctors usually view the important nodes nearby. The intersection line of the attention image and the organ contour is the place that doctors pay attention to. As Figure 3 shown, the doctor views a part of the image of a certain liver. By focusing on the intersection line of the grid and the organ contour, the contour of the part that the doctor pays attention to can be determined as shown in the right figure. Then, when there is a lesion in a similar place of the liver, the doctor may want to view the image data of a similar location. Therefore, the part that the doctor wants to view can be determined by the contour of the part that the doctor pays attention to, that is, the detection target of the present invention. Using the organ type, the lesion location, and the part that the doctor pays attention to as classification labels, the image can be sent into the neural network for learning, so as to automatically identify the locations that doctors may pay attention to near the corresponding lesion location. By labeling through the intersection line of the attention grid and the organ contour, the present invention can perform labeling automatically, greatly reducing the workload of labeling.

[0036] Step S6: Input the training samples into the neural network for training to obtain a neural network model capable of labeling the detection target.

[0037] After the processing of the previous steps, the labeled image can be obtained. The organ type and the lesion location are used to determine the approximate range that doctors may view. By focusing on the image contour, the attention image corresponding to the organ type and the lesion location can be determined. The present invention does not make specific limitations on the specific neural network results. Through the labeled samples, those skilled in the art can select different types of networks in the prior art and conduct experiments to select a better network structure.

[0038] Preferably, in order to meet the resolution requirements of different scales, the present invention uses the Stacked Hourglass Network for training (as described in "Stacked Hourglass Network for Robust Facial Landmark Localisation", Jing Yang, Qingshan Liu, Kaihua Zhang; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017, pp. 79-87).

[0039] In the present invention, sample annotation is performed by doctors when viewing medical images. Therefore, a large number of training samples can be obtained through the daily operation of the hospital. After obtaining a sufficient number of samples, the neural network can be trained. The trained model can input similar networks and output the annotation of the concerned positions.

[0040] Step S7: Annotate the lesion location and organ type of the image to be viewed, convert the annotated image into a pyramid image and perform edge detection. Input the result image of the edge detection into the trained neural network model to output the location of the detection target.

[0041] The detection image needs to be similar to the training image. Therefore, in order to perform the detection of the target location, the image also needs to be processed in a similar way to the sample before input. After the image is processed, it can be input into the model, and the model automatically annotates the location of the detection target in the contour.

[0042] Step S8: Determine the corresponding target grid according to the output location of the detection target. Organise the images corresponding to the target grid into a tree according to the pyramid image hierarchy relationship. Each node of the tree is displayed as a thumbnail. Select any tree node and display the selected node image in detail on the same interface.

[0043] Since the pyramid image has been gridified, the corresponding target grid can be determined according to the output location of the detection target. Since the pyramid image has a hierarchical relationship, the first layer (equivalent to the root of the tree) can be found by looking up from each target grid. Looking up all the target grids can obtain a tree structure. As Figure 4 shown, the neural network detects that Figure 12 on the fourth layer, Figure 3 on the third layer, and Figure 27 on the fifth layer are all positions that doctors may be concerned about. Then looking up can form a structure as Figure 4The tree structure shown. In the tree structure, each node is represented by a thumbnail, and doctors can roughly observe the outline of each node through the tree structure. When a doctor confirms that a certain node is indeed the node they want to view, selecting the corresponding node can display the corresponding node in detail. For example, Figure 4 As shown, the doctor selects the Figure 3 in the third layer, then the detailed information of the third layer Figure 3 is displayed on the left.

[0044] Through the above steps, when a doctor opens an image, it can automatically infer the possible positions the doctor may want to go to, and through simple selection operations, detailed browsing can be carried out, avoiding the time-consuming zooming operations for frequently searching for the target position.

[0045] Furthermore, when a doctor does not find the target they want to view in the tree structure, they can manually search for the target position by zooming in on the detailed browsing interface. When the target position is found, the target position can be manually added to the tree structure.

[0046] In another implementation, the present invention also provides a multi-level processing system for medical data, including the following modules:

[0047] The layering module is used to convert a medical image into a pyramid image, and the pyramid image is composed of multiple medical images with different resolutions hierarchically. The lower the layer, the higher the resolution;

[0048] The partitioning module is used to partition each layer of the pyramid image into grids. The lower the layer, the more grids there are;

[0049] The first determination module is used to record the viewing trajectory of the doctor when viewing a medical image. If no further zooming is performed to view the next layer, the grid corresponding to the current image is determined as the concerned grid;

[0050] The second determination module is used to perform edge detection on each layer of the medical pyramid image to determine the organ contour; use artificial or machine learning methods to label the organ type and lesion location on the medical image;

[0051] The annotation module is used to label the organ contour corresponding to the concerned grid of each layer of the medical pyramid image as the detection target to obtain training samples;

[0052] The training module is used to input the training samples into a neural network for training to obtain a neural network model capable of annotating the detection target;

[0053] The detection module is used to label the image to be viewed with the lesion location and organ type, convert the labeled image into a pyramid image and perform edge detection, and input the result image of the edge detection into the trained neural network model to output the location of the detection target.

[0054] The output module is used to determine the corresponding target grid according to the output location of the detection target, organize the images corresponding to the target grid into a tree according to the pyramid image hierarchy relationship, and display each node of the tree as a thumbnail.

[0055] It should be noted that the detailed implementation principle and further improvement measures of the above multi-level processing system for medical data are the same as those of the aforementioned multi-level processing method for medical data, and will not be described in detail in this embodiment. Those skilled in the art can specifically implement the multi-level processing method for medical data in the multi-level processing system for medical data according to the prior art.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the specific implementation manners of the present invention can still be modified or equivalently replaced, and any modification or equivalent replacement without departing from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

[0057] For the part of the module structure not specifically defined in the present invention, it shall be subject to the content recorded in the prior art. The prior art mentioned in the foregoing background art part and specific embodiment part of the present invention can be used as a part of the present invention to understand the meaning of some technical features or parameters. The protection scope of the present invention shall be subject to the content actually recorded in the claims.

Claims

1. A multi-level processing method for medical data, characterized in that It includes the following steps: Step S1: Convert the medical image into a pyramid image, which is hierarchically composed of multiple medical images with different resolutions, and the resolution of the bottom layer is higher; Step S2: Divide each layer of the pyramid image into grids, and the number of grids in the bottom layer is more; Step S3: When the doctor views the medical image, record the viewing trajectory of the doctor. If no further zooming is performed and the next layer is viewed, the grid corresponding to the current image is determined as the concerned grid; Step S4: Perform edge detection on each layer of the medical pyramid image to determine the organ contour; use artificial or machine learning methods to label the organ type and lesion location on the medical image; Step S5: Label the organ contour corresponding to the concerned grid of each layer of the medical pyramid image as the detection target to obtain training samples; Step S6: Input the training samples into a neural network for training to obtain a neural network model capable of labeling the detection target; Step S7: Label the lesion location and organ type of the image to be viewed, convert the labeled image into a pyramid image and perform edge detection, and input the result image of the edge detection into the trained neural network model to output the location of the detection target; Step S8: Determine the corresponding target grid according to the output location of the detection target, organize the images corresponding to the target grid into a tree according to the hierarchical relationship of the pyramid image, and display each node of the tree as a thumbnail.

2. The multi-level processing method for medical data according to claim 1, wherein: The pyramid image is divided into five layers, and the number of grids in each layer is 1, 4, 16, 64, and 256 respectively.

3. The multi-level processing method for medical data according to claim 1, characterized in that: When the time the doctor stays on a certain image is greater than the first threshold, the grid corresponding to the image is considered as the concerned grid.

4. The multi-level processing method of medical data according to claim 1, characterized in that: Select any tree node and display the selected node image in detail on the detailed viewing interface.

5. The multi-level processing method of medical data according to claim 4, wherein: When the desired target cannot be found in the tree structure, manually search for the target location by zooming in and out of the detailed viewing interface. When the target location is found, manually add the target location to the tree structure.

6. A multi-level processing system for medical data, characterized in that It includes the following modules: Hierarchical module, which is used to convert the medical image into a pyramid image, and the pyramid image is hierarchically composed of multiple medical images with different resolutions, and the resolution of the bottom layer is higher; Division module, which is used to divide each layer of the pyramid image into grids, and the number of grids in the bottom layer is more; First determination module, which is used to record the viewing trajectory of the doctor when the doctor views the medical image. If no further zooming is performed and the next layer is viewed, the grid corresponding to the current image is determined as the concerned grid; Second determination module, which is used to perform edge detection on each layer of the medical pyramid image to determine the organ contour; Use artificial or machine learning methods to label the organ type and lesion location on the medical image; Labeling module, which is used to label the organ contour corresponding to the concerned grid of each layer of the medical pyramid image as the detection target to obtain training samples; Training module, which is used to input the training samples into a neural network for training to obtain a neural network model capable of labeling the detection target; The detection module is used to label the lesion locations and organ types of the images to be viewed, convert the labeled images into pyramid images and perform edge detection, and input the result images of the edge detection into a trained neural network model to output the locations of the detection targets; The output module is used to determine the corresponding target grids according to the output locations of the detection targets, organize the images corresponding to the target grids into a tree according to the pyramid image hierarchy relationship, and display each node of the tree as a thumbnail.

7. The multi-level processing system for medical data according to claim 6, wherein: The pyramid image is divided into five layers, and the number of grids in each layer is 1, 4, 16, 64, and 256 respectively.

8. The multi-level processing system for medical data according to claim 6, wherein: Only when the doctor stays on a certain image for more than the first threshold is the grid corresponding to the image considered as the concerned grid.

9. The multi-level processing system for medical data according to claim 6, wherein: Select any tree node and display the selected node image in detail on the detailed viewing interface.

10. The multi-level processing system for medical data according to claim 9, characterized in that: When the target to be viewed cannot be found in the tree structure, manually search for the target location by zooming in and out of the detailed viewing interface. When the target location is found, manually add the target location to the tree structure.

Images