Tumor area double-view detection method and system based on image modality and view

Abstract

The invention provides a tumor area double-view detection method and system based on image modalities and views, and relates to the field of medical image analys.The method comprises the steps that multi-modal image data are collected from the body of a patient, and image data of different view angles at the same position are obtained according to time frames; extracting multi-modal features from the image data of different modalities and different view angles, and performing weighted fusion on the extracted multi-modal features to form a fused feature map; inputting the fused feature map to train and predict the 3D-CNN network so as to identify and position a tumor region, and marking the tumor region by using a mask output by the segmentation model; calculating the pixel number of the tumor area in each layer of image according to the marked mask, and multiplying the pixel number by the volume of each pixel to obtain the volume of the tumor area; and the segmentation results of each layer of image are superposed, and a three-dimensional model of a tumor region is generated by using 3D reconstruction, so that the limitation of single-mode and single-view image data in tumor detection is overcome.

Description

Technical Field

[0001] The present invention relates to the field of medical image analysis, and in particular to a dual-view detection method and system for a tumor region based on image modality and view. Background Art

[0002] In the field of medical image analysis, early detection and accurate positioning of tumors are crucial to the treatment and prognosis of patients. Traditional tumor detection methods mainly rely on single-modality image data, such as CT (computed tomography), MRI (magnetic resonance imaging), PET (positron emission tomography), etc. Although these methods have been widely used in clinical practice, due to the limitations of viewing angles, image noise, and incomplete consistency of information between different modal images, single-modality detection cannot provide sufficient accuracy and comprehensiveness in some cases.

[0003] The single-view detection method uses image data from a single perspective (such as axial, coronal or sagittal view) for tumor detection and localization. This method has problems such as perspective limitation, image noise and information asymmetry. Among them, single-view images can only provide specific slice information of the image, and it is easy to miss key information in other perspectives, resulting in a decrease in the detection rate of tumors; medical images are usually accompanied by various noises, such as motion artifacts, equipment noise, etc., which will affect the accuracy of detection; the information provided by image data of different modalities is complementary, but single-modality images cannot fully utilize the information of other modalities for more comprehensive analysis.

[0004] Therefore, in view of the problem that the traditional single-view detection method has insufficient detection accuracy due to viewing angle limitation and image noise, it is urgent to propose a dual-view detection method and system for tumor areas based on imaging modality and view, which can improve the accuracy and robustness of tumor detection by integrating multimodal imaging information and dual-view technology. Summary of the invention

[0005] In view of this, in order to solve the problem of insufficient detection accuracy of traditional single-view detection methods due to viewing angle limitations and image noise, the purpose of the present invention is to propose a dual-view detection method and system for tumor areas based on imaging modality and view, aiming to improve the accuracy and robustness of tumor detection and overcome the limitations of single modality and single-view image data in tumor detection.

[0006] To achieve the above object, the present invention provides the following technical solutions: Based on the above objectives, in a first aspect, the present invention provides a dual-view detection method for a tumor region based on imaging modalities and views, comprising the following steps: Collect multimodal imaging data from the patient's body and obtain imaging data of the same position at different viewing angles according to time frames; Perform registration and noise removal preprocessing on the collected image data, and extract multimodal features from image data of different modalities and different perspectives; According to multi-modality and dual-view, the extracted multi-modal features are weighted fused through the attention mechanism to form a fused feature map; The fused feature map is input to train and predict the 3D-CNN network to identify and locate the tumor area; Use the mask output by the segmentation model to annotate the tumor region; The number of pixels in the tumor area in each layer of the image is calculated according to the annotated mask, and the volume of the tumor area is calculated by multiplying it by the volume of each pixel; The segmentation results of each layer of images are superimposed, and 3D reconstruction is used to generate a three-dimensional model of the tumor area for visual display.

[0007] As a further embodiment of the present invention, multimodal imaging data are collected from the patient's body, including imaging data of axial views, coronal views and sagittal views provided by CT (computed tomography), MRI (magnetic resonance imaging), and PET (positron emission tomography), as well as two-dimensional cross-sectional image data of ultrasound imaging examination.

[0008] As a further solution of the present invention, when acquiring image data of different perspectives at the same position according to a time frame, the imaging device is synchronized and calibrated; utilizing a synchronous trigger mechanism, the same anatomical position is located by placing marked markers, so that the imaging device acquires image data at the same position point, and the acquired image data is divided according to time frames so that each time frame corresponds to an image of the same anatomical position, and image acquisition modes of different perspectives, such as axial, coronal, and sagittal, are used to acquire data from multiple directions, thereby acquiring image data of different perspectives at the same position.

[0009] As a further solution of the present invention, when performing CT, MRI and ultrasound imaging examinations, the patient is scanned in layers using a CT device to obtain images of different time frames; the same position is scanned at multiple angles using an MRI device to obtain images of different time frames; and the target area is imaged at multiple angles using an ultrasound device to record changes in different time frames.

[0010] As a further solution of the present invention, when registering the collected image data, an image of one time frame is selected as the reference image, and the images of the remaining time frames are selected as the images to be registered; based on the feature point registration, feature points of the reference image and the image to be registered are extracted, and the transformation matrix of the feature points is calculated so that the image to be registered is aligned with the reference image in space, wherein the calculation formula of the transformation matrix T of the feature points is:

[0011] In the formula, and ,in, is the first The location of feature points; is the image to be registered (target image) that is the first The position of the feature points corresponding to the feature points, Represents the application of transformation matrix After that, the first The position of feature points in the image to be registered; among them, The expression of the parameters to minimize the expression makes the sum of square errors between feature points minimum; among them, is the error metric, which represents the The location of the feature points And the corresponding feature points in the image to be registered The square of the Euclidean distance between them.

[0012] As a further solution of the present invention, when the extracted multimodal features are weightedly fused through the attention mechanism, images of different modalities and perspectives are aligned in the same coordinate system, and multimodal features of texture, shape and edge are extracted from the preprocessed and registered images, and the multimodal features are weightedly fused using the attention mechanism to form a fused feature map.

[0013] As a further solution of the present invention, when the tumor area is identified and located, the new multimodal data is input into the fusion module to generate a fusion feature map, and then input into the 3D-CNN network for tumor identification and location.

[0014] As a further solution of the present invention, when the tumor area is annotated using the mask output by the segmentation model, the medical image containing the tumor area is segmented and extracted, and the binary mask output by the segmentation model, each pixel of the binary mask takes a value of 0 or 1, where "1" represents the tumor area and "0" represents the background; the contour of the tumor area in the mask is extracted by edge detection, the extracted contour is drawn on the original medical image, and the annotated image is saved.

[0015] As a further solution of the present invention, when calculating the volume of the tumor area, the annotation mask and pixel spacing of the tumor area are obtained, and the number of pixels and the volume of each pixel in the tumor area in each layer of the image are calculated.

[0016] As a further solution of the present invention, the annotation mask and pixel spacing of the tumor area are obtained. When the mask is annotated, a binary mask is obtained after the tumor area is annotated, and each layer of the image has a mask. The actual spatial size of each pixel is used as the pixel spacing, wherein the pixel spacing is the pixel size of the cross section, and the unit is millimeter; the distance between image layers is the layer thickness, and the unit is millimeter.

[0017] As a further solution of the present invention, when calculating the number of pixels in the tumor area in each layer of the image, the binary mask of each layer of the image is input and the pixels are counted, the number of tumor areas in each layer of the mask is counted, and when calculating the volume of each pixel, the volume of each pixel is obtained by calculating the pixel spacing and layer thickness.

[0018] As a further embodiment of the present invention, when using 3D reconstruction to generate a three-dimensional model of the tumor area, the following steps are included: Reading the segmentation mask of each layer of image to form three-dimensional voxel data, wherein reading the segmentation mask includes superimposing the binary mask of each layer of image in a third dimension to form three-dimensional voxel data; Generating an isosurface from the three-dimensional voxel data using a Marching Cubes algorithm, wherein the isosurface comprises generating a three-dimensional surface mesh from the voxel data; The mesh coordinates are converted to real-space coordinates, where the real-space coordinates include adjusting the mesh vertex coordinates according to pixel spacing and layer thickness.

[0019] As a further solution of the present invention, when forming three-dimensional voxel data, the segmentation mask of each layer of the image is read to form a three-dimensional array M (x, y, z), and a 3D voxel array V (x, y, z) with the same shape as the input mask is created. According to the value of each layer of the mask, the voxels at the corresponding positions are set to 1 to represent the tumor area, and the remaining positions are set to 0, and the 2D segmentation results are superimposed to generate 3D voxel data.

[0020] In a second aspect, the present invention provides a dual-view detection system for a tumor region based on imaging modalities and views, comprising: Data acquisition module: used to collect multimodal imaging data from the patient's body and obtain imaging data of different viewing angles at the same position according to the time frame; Feature extraction module: used to extract multimodal features from image data of different modalities and different perspectives; Feature fusion module: used to perform weighted fusion of the extracted multimodal features based on multimodality and dual views through the attention mechanism to form a fused feature map; Training and prediction module: used to input the fused feature map to train and predict the 3D-CNN network to identify and locate the tumor area; Segmentation and annotation module: used to annotate the tumor area using the mask output by the segmentation model; Volume calculation module: used to calculate the number of pixels in the tumor area in each layer of the image according to the marked mask, and multiply the volume of each pixel to calculate the volume of the tumor area; 3D reconstruction module: used to superimpose the segmentation results of each layer of images and use 3D reconstruction to generate a three-dimensional model of the tumor area.

[0021] As a further solution of the present invention, the dual-view detection system for tumor areas based on imaging modalities and views also includes a data preprocessing module, which includes an image registration unit and a noise removal unit; the image registration unit is used to perform spatial registration on the collected multimodal imaging data to ensure that the imaging data of different modalities and viewing angles are aligned, and the noise removal unit is used to remove noise in the imaging data to improve image quality.

[0022] In another aspect of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, any one of the above-mentioned methods for detecting tumor areas based on imaging modalities and views according to the present invention is executed.

[0023] In yet another aspect of the present invention, a computer-readable storage medium is provided, storing computer program instructions, which, when executed, implement any of the above-mentioned methods for detecting tumor regions based on imaging modalities and views according to the present invention.

[0024] Compared with the prior art, the dual-view detection method and system for tumor regions based on imaging modalities and views proposed in the present invention have the following beneficial effects: 1. Improved diagnostic accuracy through multimodal image fusion and viewing angle diversity. The present invention can capture more pathological information and improve the comprehensiveness and accuracy of diagnosis by integrating image data of multiple modalities such as CT, MRI, and PET. Image data from different viewing angles can provide more comprehensive tumor information, which helps to accurately locate and identify tumor areas.

[0025] 2. Through attention mechanism fusion and 3D-CNN network training, misdiagnosis and missed diagnosis are reduced. The present invention uses the attention mechanism to perform weighted fusion of multi-modal and multi-view features, enhances the ability to capture important features, and reduces the probability of misdiagnosis and missed diagnosis; through 3D convolutional neural network training and prediction of fused features, it can more accurately identify and locate tumor areas and improve segmentation accuracy.

[0026] 3. Improve image processing efficiency through automated processing flow and data processing. The present invention highly automates the entire process from data acquisition, preprocessing, feature extraction and fusion to 3D modeling and visualization, reducing manual intervention, improving processing efficiency and consistency, and using image registration, noise removal, feature extraction and fusion to ensure the system's high efficiency in processing large-scale image data.

[0027] 4. Through accurate tumor segmentation and annotation, as well as volume calculation and 3D reconstruction, accurate treatment support is provided. The present invention uses the segmentation model to accurately segment and annotate the tumor area, generate high-quality segmentation masks, and provide a reliable basis for subsequent clinical decision-making; by accurately calculating the tumor volume and generating a three-dimensional model, it helps doctors to carry out detailed preoperative planning and postoperative tracking.

[0028] 5. Wide clinical application, applicable to various tumor types, and cross-platform compatibility. The present invention can be applied to the detection and segmentation of tumors of various types and locations, has broad clinical application prospects, can be integrated into a variety of medical image processing platforms, and enhances the applicability and flexibility of the system. By improving the accuracy and precision of tumor detection, the system helps to detect tumors early and intervene in time, improving the prognosis of patients. Based on accurate tumor segmentation and three-dimensional reconstruction models, doctors can formulate more personalized treatment plans and improve treatment effects.

[0029] In summary, the dual-view detection method and system for tumor areas of the present invention significantly improves the accuracy, efficiency and clinical application value of tumor detection and segmentation through the fusion processing of multi-modal and multi-view imaging data, combined with advanced image processing and deep learning technology, and has broad medical application prospects and market potential.

[0030] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the exemplary embodiments or related technical descriptions. The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 The figure is a flow chart of a dual-view detection method for a tumor region based on imaging modalities and views according to an embodiment of the present invention.

[0032] Figure 2The present invention is a flowchart of generating a three-dimensional model of a tumor region in a dual-view detection method of a tumor region based on imaging modalities and views according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, system, product or device that includes a series of steps or units.

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0038] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0039] Aiming at the problem of insufficient detection accuracy of traditional single-view detection methods due to viewing angle limitation and image noise, the present invention proposes a dual-view detection method and system for tumor areas based on imaging modality and view, which improves the accuracy and robustness of tumor detection by integrating multimodal imaging information and dual-view technology.

[0040] See also Figure 1As shown, an embodiment of the present invention provides a dual-view detection method for a tumor region based on an imaging modality and a view, the method comprising the following steps: Step S10: Collect multimodal image data from the patient's body, and obtain image data of different viewing angles at the same position according to a time frame.

[0041] In this step, multimodal imaging data are collected from the patient's body, including axial, coronal and sagittal image data provided by CT (computed tomography), MRI (magnetic resonance imaging), and PET (positron emission tomography), as well as two-dimensional cross-sectional image data from ultrasound imaging.

[0042] Among them, when acquiring image data of different perspectives at the same position according to the time frame, the imaging equipment (CT, MRI, PET, ultrasound, etc.) is synchronized and calibrated; the same anatomical position is located by marking the placement of markers using a synchronous trigger mechanism, so that the imaging equipment collects image data at the same position point, and the collected image data is divided according to the time frame so that each time frame corresponds to the image of the same anatomical position, and the image acquisition modes of different perspectives, such as axial, coronal, and sagittal, are used to obtain data from multiple directions, so as to obtain image data of different perspectives at the same position.

[0043] When performing CT, MRI and ultrasound imaging examinations, CT equipment is used to perform layered scans on the patient to obtain images of different time frames; MRI equipment is used to perform multi-angle scans on the same location to obtain images of different time frames; and ultrasound equipment is used to perform multi-angle imaging of the target area to record changes in different time frames.

[0044] In this step, by integrating imaging data from multiple modalities such as CT, MRI, and PET, more pathological information can be captured, improving the comprehensiveness and accuracy of the diagnosis. Image data from different perspectives can provide more comprehensive tumor information, which helps to accurately locate and identify tumor areas. Multimodal image fusion and perspective diversity can improve diagnostic accuracy.

[0045] Step S20: perform registration and noise removal preprocessing on the collected image data, and extract multimodal features from image data of different modalities and different viewing angles.

[0046] In this step, when registering the collected image data, an image of one time frame is selected as the reference image, and the images of the remaining time frames are selected as the images to be registered; based on the feature point registration, the feature points of the reference image and the image to be registered are extracted, and the transformation matrix of the feature points is calculated so that the image to be registered is aligned with the reference image in space. The calculation formula of the transformation matrix T of the feature points is:

[0047] In the formula, and ,in, is the first The location of feature points; is the image to be registered (target image) that is the first The position of the feature points corresponding to the feature points, Represents the application of transformation matrix After that, the first The position of feature points in the image to be registered; among them, The expression of the parameters to minimize the expression makes the sum of square errors between feature points minimum; among them, is the error metric, which represents the The location of the feature points And the corresponding feature points in the image to be registered The square of the Euclidean distance between them.

[0048] For example, if a patient needs to acquire multimodal images of the heart in order to observe changes in the heart at different time frames, then the registration and noise removal preprocessing of the acquired image data includes acquiring CT and MRI data and data processing, wherein first, a CT scan is performed on the patient to obtain multiple slice images of the heart; then, an MRI scan is performed on the patient to obtain images of the heart from different perspectives. The CT and MRI images are calibrated using image registration so that they are aligned in the same coordinate system, and then the CT and MRI data are fused to obtain a comprehensive, multi-perspective heart image.

[0049] Step S30: According to the multi-modality and dual-view, the extracted multi-modal features are weighted fused through the attention mechanism to form a fused feature map.

[0050] In this step, when the extracted multimodal features are weightedly fused through the attention mechanism, images of different modalities and perspectives are aligned in the same coordinate system, and multimodal features of texture, shape and edge are extracted from the preprocessed and registered images. The multimodal features are weightedly fused using the attention mechanism to form a fused feature map.

[0051] Step S40: Input the fused feature map to train and predict the 3D-CNN network to identify and locate the tumor area.

[0052] In this step, when the tumor area is identified and located, the new multimodal data is input into the fusion module to generate a fusion feature map, which is then input into the 3D-CNN network for tumor identification and location.

[0053] For example, if liver tumor detection and segmentation is performed, in the data acquisition and preprocessing stage, multimodal image data containing liver tumors are collected, for example: CT imaging: provides high-resolution anatomical information of the liver.

[0054] MRI images: provide detailed information about the soft tissues of the liver.

[0055] PET imaging: provides information on the metabolic activity of tumors.

[0056] Then, in the data preprocessing stage, the CT, MRI, and PET images are registered and aligned in the same coordinate system, the images are denoised using filtering, and the image data is standardized so that the image data of different modalities have consistent scales and ranges; then multimodal feature extraction and fusion are performed to extract multimodal features of texture, shape, and edge from the preprocessed and registered images, where: CT images: Extract features of changes in tissue density.

[0057] MRI images: Extract the morphological features of liver soft tissue.

[0058] PET images: Extract features of metabolically active areas of tumors.

[0059] Then, the feature maps of different modalities are weighted, focusing on capturing the features that contribute most to tumor detection. The multimodal features are weightedly fused through the attention mechanism to form a fused feature map with comprehensive information.

[0060] Then, in the 3D-CNN network training and prediction stage, the fused feature map is input into the 3D convolutional neural network, and the labeled tumor region data is used to train the 3D-CNN, optimize the network parameters, and predict the fused feature map through the 3D-CNN network when new multimodal data is input, identify and locate the tumor region. Finally, the results are verified and analyzed, and the recognition accuracy of the system is verified by comparing the system recognition results with the expert annotation standards, integrating different modal image data, providing more comprehensive tumor information, improving diagnostic accuracy, and supplementing image data from different perspectives to ensure the accuracy of tumor region recognition.

[0061] This embodiment uses the attention mechanism to perform weighted fusion of multi-modal and multi-view features, thereby enhancing the ability to capture important features and reducing the probability of misdiagnosis and missed diagnosis. By training and predicting the fused features through a 3D convolutional neural network, the tumor area can be more accurately identified and located, and the segmentation accuracy is improved. Through attention mechanism fusion and 3D-CNN network training, misdiagnosis and missed diagnosis are reduced.

[0062] Step S50: Use the mask output by the segmentation model to mark the tumor area.

[0063] In this step, when the mask output by the segmentation model is used to annotate the tumor area, the medical image containing the tumor area is segmented and extracted. The binary mask output by the segmentation model has a value of 0 or 1 for each pixel, where "1" represents the tumor area and "0" represents the background. The contour of the tumor area in the mask is extracted by edge detection, the extracted contour is drawn on the original medical image, and the annotated image is saved.

[0064] For example, if there is a CT image with a resolution of 512x512 pixels, the binary mask generated by the segmentation model is as follows: 0 0 0 1 1 0 0 0 0 0 1 1 1 1 0 0 0 1 1 1 1 1 1 0 0 0 0 1 1 0 0 0 Use Canny edge detection to extract the edges and draw the outline on the original image and save it as "marked_image.png".

[0065] Step S60: Calculate the number of pixels in the tumor area in each layer of the image according to the marked mask, and multiply it by the volume of each pixel to obtain the volume of the tumor area.

[0066] In this step, when calculating the volume of the tumor area, the annotation mask and pixel spacing of the tumor area are obtained, and the number of pixels in the tumor area in each layer of the image and the volume of each pixel are calculated. The annotation mask and pixel spacing of the tumor area are obtained. When annotating the mask, the tumor area is annotated with a binary mask, and each layer of the image has a mask; the actual spatial size of each pixel is used as the pixel spacing, where the pixel spacing is the pixel size of the cross section, in millimeters; the distance between image layers is the layer thickness, in millimeters.

[0067] In this embodiment, when calculating the number of pixels in the tumor area in each layer of the image, the binary mask of each layer of the image is input and the pixels are counted, the number of tumor areas in each layer of the mask is counted, and when calculating the volume of each pixel, the volume of each pixel is obtained by calculating the pixel spacing and layer thickness.

[0068] For example, when calculating the volume of the tumor area, assuming that there are 100 "1"s in the mask of a layer of image, the pixel spacing is 0.5 mm, and the layer thickness is 1 mm, then: the volume of each pixel = 0.5 2 *1=0.25 cubic millimeters; the volume of the tumor area in this layer = 100*0.25=25 cubic millimeters.

[0069] Step S70: superimpose the segmentation results of each layer of the image, use 3D reconstruction to generate a three-dimensional model of the tumor area and perform visual display.

[0070] In this step, see Figure 2 As shown, when using 3D reconstruction to generate a three-dimensional model of the tumor area, the following steps are included: Step S701, reading the segmentation mask of each layer of image to form three-dimensional voxel data, wherein reading the segmentation mask includes superimposing the binary mask of each layer of image in the third dimension to form three-dimensional voxel data; Step S702: using the Marching Cubes algorithm to generate an isosurface from the three-dimensional voxel data, wherein the isosurface includes generating a three-dimensional surface mesh from the voxel data; Step S703: converting the grid coordinates into real space coordinates, wherein the real space coordinates include adjusting the grid vertex coordinates according to the pixel spacing and the layer thickness.

[0071] When forming three-dimensional voxel data, the segmentation mask of each layer of the image is read to form a three-dimensional array M (x, y, z), and a 3D voxel array V (x, y, z) with the same shape as the input mask is created. According to the value of each layer of the mask, the voxels at the corresponding positions are set to 1 to represent the tumor area, and the remaining positions are set to 0. The 2D segmentation results are superimposed to generate 3D voxel data.

[0072] This embodiment uses a segmentation model to accurately mark the tumor area in the medical image, calculate the tumor volume, and generate a three-dimensional reconstruction model, thereby providing detailed tumor morphology and volume information. The method of the present invention has high accuracy and repeatability, can assist doctors in diagnosis and treatment planning, and provide better medical services for patients.

[0073] The dual-view detection method of tumor regions based on imaging modalities and views of the present invention is highly automated from data acquisition, preprocessing, feature extraction and fusion to 3D modeling and visualization, which reduces manual intervention, improves processing efficiency and consistency, and uses image registration, noise removal, feature extraction and fusion to ensure the efficiency of the system when processing large-scale imaging data. The present invention uses a segmentation model to accurately segment and annotate the tumor region, generate high-quality segmentation masks, and provide a reliable basis for subsequent clinical decision-making; by accurately calculating the tumor volume and generating a three-dimensional model, it helps doctors to carry out detailed preoperative planning and postoperative tracking.

[0074] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0075] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0076] In a second aspect of the embodiments of the present invention, the present invention further provides a dual-view detection system for a tumor region based on imaging modalities and views, comprising: Data acquisition module: used to collect multimodal imaging data from the patient's body and obtain imaging data of different viewing angles at the same position according to the time frame; Feature extraction module: used to extract multimodal features from image data of different modalities and different perspectives; Feature fusion module: used to perform weighted fusion of the extracted multimodal features based on multimodality and dual views through the attention mechanism to form a fused feature map; Training and prediction module: used to input the fused feature map to train and predict the 3D-CNN network to identify and locate the tumor area; Segmentation and annotation module: used to annotate the tumor area using the mask output by the segmentation model; Volume calculation module: used to calculate the number of pixels in the tumor area in each layer of the image according to the marked mask, and multiply the volume of each pixel to calculate the volume of the tumor area; 3D reconstruction module: used to superimpose the segmentation results of each layer of images and use 3D reconstruction to generate a three-dimensional model of the tumor area.

[0077] Among them, the dual-view detection system for tumor areas based on imaging modalities and views also includes a data preprocessing module, which includes an image registration unit and a noise removal unit; the image registration unit is used to perform spatial registration on the collected multimodal imaging data to ensure that the imaging data of different modalities and viewing angles are aligned, and the noise removal unit is used to remove noise in the imaging data to improve image quality.

[0078] Through the above detailed steps and code examples, the dual-view detection system of the tumor area based on imaging modality and view of the present invention is used to execute the steps of the dual-view detection method of the tumor area based on imaging modality and view in the above embodiment, which will not be repeated here.

[0079] The dual-view detection system of tumor areas based on imaging modalities and views of the present invention can be applied to the detection and segmentation of tumors of various types and locations, has broad clinical application prospects, and can be integrated into a variety of medical image processing platforms, thereby enhancing the applicability and flexibility of the system. By improving the accuracy and precision of tumor detection, the system helps to detect tumors at an early stage and conduct timely intervention, thereby improving the prognosis of patients. Based on accurate tumor segmentation and three-dimensional reconstruction models, doctors can formulate more personalized treatment plans and improve treatment effects.

[0080] The dual-view detection method and system for tumor areas of the present invention significantly improve the accuracy, efficiency and clinical application value of tumor detection and segmentation through the fusion processing of multi-modal and multi-view imaging data, combined with advanced image processing and deep learning technology, and has broad medical application prospects and market potential.

[0081] According to a third aspect of an embodiment of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of any one of the above embodiments is implemented.

[0082] The computer device includes a processor and a memory, and may also include an input system and an output system. The processor, memory, input system and output system may be connected via a bus or other means, and the input system may receive input digital or character information, and generate signal input related to the migration of the tumor area dual view detection based on the imaging modality and view. The output system may include a display device such as a display screen.

[0083] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the dual-view detection method for tumor regions based on imaging modalities and views in the embodiments of the present application. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of the dual-view detection method for tumor regions based on imaging modalities and views, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0084] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data. The processors of multiple computer devices of the computer device of this embodiment execute various functional applications and data processing of the server by running non-volatile software programs, instructions and modules stored in the memory, that is, implementing the steps of the dual-view detection method of the tumor area based on the imaging modality and view of the above-mentioned method embodiment.

[0085] It should be understood that, to the extent that they do not conflict with each other, all the embodiments, features and advantages described above for the dual-view detection method for tumor regions based on imaging modalities and views according to the present invention are also applicable to the dual-view detection and storage medium for tumor regions based on imaging modalities and views according to the present invention.

[0086] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0087] Finally, it should be noted that the computer-readable storage medium (e.g., memory) herein may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. As an example and not by way of limitation, a non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. A volatile memory may include a random access memory (RAM), which may act as an external cache memory. As an example and not by way of limitation, RAM may be obtained in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

[0088] The various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein may be implemented or executed using the following components designed to perform the functions herein: a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and / or any other such configuration.

[0089] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0090] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0091] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A dual-view detection method for tumor regions based on imaging modalities and views, characterized in that: The method comprises the following steps: Collect multimodal imaging data from the patient's body and obtain imaging data of the same position at different viewing angles according to time frames; Perform registration and noise removal preprocessing on the collected image data, and extract multimodal features from image data of different modalities and different perspectives; According to multi-modality and dual-view, the extracted multi-modal features are weighted fused through the attention mechanism to form a fused feature map; The fused feature map is input to train and predict the 3D-CNN network to identify and locate the tumor area; Use the mask output by the segmentation model to annotate the tumor region; The number of pixels in the tumor area in each layer of the image is calculated according to the annotated mask, and the volume of the tumor area is calculated by multiplying it by the volume of each pixel; The segmentation results of each layer of images are superimposed, and 3D reconstruction is used to generate a three-dimensional model of the tumor area for visual display.

2. The dual-view tumor region detection method based on imaging modality and view according to claim 1, characterized in that: The multimodal imaging data collected from the patient's body include axial, coronal and sagittal image data provided by CT, MRI and PET, as well as two-dimensional cross-sectional image data from ultrasound imaging examinations.

3. The dual-view detection method for tumor regions based on imaging modalities and views as claimed in claim 2, characterized in that: When acquiring image data of different perspectives at the same position according to the time frame, the imaging device is synchronized and calibrated; utilizing the synchronization trigger mechanism, the same anatomical position is located by placing marked markers, so that the imaging device acquires image data at the same position point, and the acquired image data is divided according to the time frame so that each time frame corresponds to the image of the same anatomical position, and using image acquisition modes of different perspectives, such as axial, coronal, and sagittal, to acquire data from multiple directions, thereby acquiring image data of different perspectives at the same position.

4. The dual-view tumor region detection method based on imaging modality and view according to claim 3, characterized in that: When performing CT, MRI and ultrasound imaging examinations, use CT equipment to perform layered scans on the patient to obtain images in different time frames; use MRI equipment to perform multi-angle scans on the same location to obtain images in different time frames; use ultrasound equipment to perform multi-angle imaging of the target area and record changes in different time frames.

5. The dual-view tumor region detection method based on imaging modality and view according to claim 1, characterized in that: When registering the collected image data, an image of one time frame is selected as the reference image, and the images of the remaining time frames are selected as the images to be registered. The feature points of the reference image and the image to be registered are extracted based on feature point registration, and the transformation matrix of the feature points is calculated so that the image to be registered is aligned with the reference image in space. The calculation formula of the transformation matrix T of the feature points is: ； In the formula, and ,in, The first The location of feature points; is the image to be registered that is the first The position of the feature points corresponding to the feature points, Represents the application of transformation matrix After that, the first The position of feature points in the image to be registered; among them, The expression of the parameters to minimize the expression makes the sum of square errors between feature points minimum; among them, is the error metric, which represents the The location of the feature points And the corresponding feature points in the image to be registered The square of the Euclidean distance between them.

6. The dual-view tumor region detection method based on imaging modality and view according to claim 1, characterized in that: When the extracted multimodal features are weightedly fused through the attention mechanism, images of different modalities and perspectives are registered and aligned in the same coordinate system, and multimodal features of texture, shape and edge are extracted from the preprocessed and registered images. The multimodal features are weightedly fused using the attention mechanism to form a fused feature map.

7. The dual-view tumor region detection method based on imaging modality and view according to claim 1, characterized in that: When the mask output by the segmentation model is used to annotate the tumor area, the medical image containing the tumor area is segmented and extracted. The binary mask output by the segmentation model has a value of 0 or 1 for each pixel, where "1" represents the tumor area and "0" represents the background. The contour of the tumor area in the mask is extracted through edge detection, and the extracted contour is drawn on the original medical image, and the annotated image is saved.

8. The dual-view detection method for tumor regions based on imaging modalities and views according to claim 7, characterized in that: When calculating the volume of the tumor region, the annotation mask and pixel spacing of the tumor region are obtained, and the number of pixels and the volume of each pixel in the tumor region in each layer of the image are calculated.

9. The dual-view tumor region detection method based on imaging modality and view according to claim 8, characterized in that: When obtaining the annotation mask and pixel spacing of the tumor area, the binary mask after annotating the tumor area and a mask for each layer of the image are obtained to obtain the annotation mask, and the pixel spacing is determined according to the actual spatial size of each pixel; wherein the pixel spacing is the pixel size in the cross section, and the distance between the image layers is the layer thickness.

10. A dual-view detection system for tumor regions based on imaging modalities and views, characterized in that: The system is used to perform the dual-view detection method for a tumor region based on imaging modalities and views as described in any one of claims 1 to 9, the system comprising: Data acquisition module: used to collect multimodal imaging data from the patient's body and obtain imaging data of different viewing angles at the same position according to the time frame; Feature extraction module: used to extract multimodal features from image data of different modalities and different perspectives; Feature fusion module: used to perform weighted fusion of the extracted multimodal features based on multimodality and dual views through the attention mechanism to form a fused feature map; Training and prediction module: used to input the fused feature map to train and predict the 3D-CNN network to identify and locate the tumor area; Segmentation and annotation module: used to annotate the tumor area using the mask output by the segmentation model; Volume calculation module: used to calculate the number of pixels in the tumor area in each layer of the image according to the marked mask, and multiply the volume of each pixel to calculate the volume of the tumor area; 3D reconstruction module: used to superimpose the segmentation results of each layer of images and use 3D reconstruction to generate a three-dimensional model of the tumor area.

Images