Medical image target identification method, device, equipment and medium

Abstract

The present application discloses a method, apparatus, device, and medium for medical image target identification. The method comprises: obtaining a pair of images to be predicted, the pair comprising a corresponding first image and a second image, wherein the lesion in the first image is annotated; determining Euclidean transformation parameters based on the first image based on a spatial transformation network; performing spatial transformation and interpolation processing on the first image based on the Euclidean transformation parameters to obtain a first region of interest, and performing spatial transformation and cropping processing on the second image to obtain a second region of interest; performing feature extraction on the first region of interest and the second region of interest based on a feature extraction network to obtain a first feature and a second feature; fusing the first feature and the second feature based on a classification prediction network to obtain a fused feature, and classifying the fused feature to obtain a target identification result. The present application significantly improves the accuracy of target identification in medical images.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a method, device, equipment and medium for identifying medical image targets. Background Art

[0002] With the development of medical image processing technology, automatic object recognition is increasingly used in medical image object recognition and identification. In medical images, the reason why lesions can be identified is not only based on size or edge features, but more on the density difference between the area and the surrounding areas.

[0003] However, in the prior art, when medical image processing is performed, a large amount of subtle structural information in the original image is lost, so that the tissue structure and its adjacent relationships cannot be accurately displayed, and ultimately the lesions are covered up or missed. Summary of the Invention

[0004] In response to the above situation, the embodiments of the present application propose a medical image target identification method, device, equipment and medium to overcome or partially overcome the shortcomings of the existing technology.

[0005] In a first aspect, embodiments of the present application disclose a method for identifying medical image targets. The method is implemented using a medical image target identification model, wherein the model includes a spatial transformation network, a feature extraction network, and a classification prediction network connected in sequence; the method includes:

[0006] Acquire a pair of images to be predicted, the pair comprising a corresponding first image and a second image, wherein a lesion in the first image is labeled;

[0007] Determining Euclidean transformation parameters based on the first image based on the spatial transformation network; performing spatial transformation and interpolation processing on the first image based on the Euclidean transformation parameters to obtain a first region of interest; and performing spatial transformation and cropping processing on the second image based on the Euclidean transformation parameters to obtain a second region of interest;

[0008] Based on the feature extraction network, extracting features from the first region of interest and the second region of interest respectively to obtain a first feature and a second feature;

[0009] Based on the classification prediction network, the first feature and the second feature are fused to obtain a fused feature, and the fused feature is classified to obtain a target identification result.

[0010] In a second aspect, an embodiment of the present application further provides a medical image target identification device, wherein the device deploys a medical image target identification model, wherein the model includes a spatial transformation network, a feature extraction network, and a classification prediction network connected in sequence; the device includes:

[0011] an acquisition unit, configured to acquire a pair of images to be predicted, the pair of images to be predicted comprising a corresponding first image and a second image, wherein a lesion in the first image is marked;

[0012] a positioning unit, configured to determine Euclidean transformation parameters based on the spatial transformation network and the first image; perform spatial transformation and interpolation processing on the first image according to the Euclidean transformation parameters to obtain a first region of interest; and perform spatial transformation and cropping processing on the second image according to the Euclidean transformation parameters to obtain a second region of interest;

[0013] an extraction unit, configured to perform feature extraction on the first region of interest and the second region of interest respectively based on a feature extraction network to obtain a first feature and a second feature;

[0014] A classification unit is used to fuse the first feature and the second feature based on the classification prediction network to obtain a fused feature, and classify the fused feature to obtain a target identification result.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor; and a memory arranged to store computer-executable instructions, wherein when the executable instructions are executed, the processor executes the above-mentioned medical image target identification method.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the above-mentioned medical image target identification method.

[0017] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0018] The present application designs a medical image target identification model, which includes a spatial transformation network, a feature extraction network and a classification prediction network connected in sequence; when the image to be predicted is predicted and classified, images of the same part of the same patient are stored separately, one of which has the lesion manually or automatically outlined and annotated in advance, and then the first image and the second image are input into the medical image target identification model respectively, and the two first enter the spatial transformation network, and the Euclidean transformation parameters of the first image are learned through the spatial transformation network, and the first image is spatially transformed and interpolated according to the learned Euclidean transformation parameters to obtain a first region of interest in which the lesion in the first image is enhanced, thereby realizing the use of linear interpolation. The region of interest in the first image is restored by a value method, and the restored image is used for back propagation learning, so that the lesion features in the first image are enhanced; for the second image, it is cropped according to the Euclidean transformation parameters learned from the first image to obtain a second region of interest with a suitable position and no information loss; then the first region of interest and the second region of interest are sent to the feature extraction network, and their information is extracted respectively to obtain the first feature and the second feature respectively; finally, the first feature and the second feature are sent to the classification prediction network, and the first feature and the second feature are fused by feature fusion to obtain the fused feature, and the fused feature is classified to obtain the target identification result.

[0019] This application addresses the prior art, in which the attention mechanism based on spatial transformation uses an interpolation method to generate a region of interest image to ensure the continuity of the network's reverse gradient transfer. This operation will result in loss of detail in the generated image and may cause loss of information of some key diagnostic features, thereby leading to missed diagnosis or misdiagnosis. This application proposes a mechanism for restoring the original image with the addition of a region of interest. Two images of the same affected area are cleverly used, one of which is annotated and the Euclidean transformation parameters are learned. The region of interest in the first image is restored using linear interpolation. The restored image is used for backpropagation learning, so that the lesion features in the first image are enhanced, and the Euclidean transformation parameters are learned. The learned Euclidean transformation parameters are applied to the cropping of the other image to obtain a region of interest without any information loss, thereby obtaining multiple features of the target lesion without loss, thereby ensuring that the image detail information in the original image is not lost, and significantly improving the accuracy of target identification in medical images. In addition, this application divides the overall task into two subtask branches, redistributing the network's learning ability, thereby reducing the difficulty of the original task and making the network model easier to train. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1-a An image of a small lung nodule obtained by CT scan is shown;

[0022] Figure 1-b The STN algorithm according to the prior art is shown. Figure 1-a An image of a small lung nodule obtained by image processing;

[0023] Figure 1-c The interpolation algorithm according to the prior art is shown. Figure 1-a An image of a small lung nodule obtained by image processing;

[0024] Figure 2 A schematic diagram of a process for identifying medical image targets according to an embodiment of the present application is shown;

[0025] Figure 3 A schematic diagram of the structure of a medical image target identification model according to an embodiment of the present application is shown;

[0026] Figure 4 A schematic diagram showing the flow of data in a medical image target identification model according to an embodiment of the present application is shown;

[0027] Figure 5 A schematic structural diagram of a medical image target identification device according to an embodiment of the present application is shown;

[0028] Figure 6 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0029] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0031] CT scanning is an important medical imaging technology. Doctors can diagnose diseases by distinguishing characteristics such as the CT values ​​(HU values) of lesions and normal tissue. However, in the human body, the CT values ​​of normal and lesioned tissue sometimes differ very slightly (3 to 5 HU). In such cases, a subtle density difference may indicate a specific lesion. When diagnosing a CT image, doctors first observe the image as a whole and then select specific areas of focus, focusing their attention on these areas to obtain more detailed information and reduce the interference of unnecessary information.

[0032] The visual attention mechanism in deep learning essentially mimics human visual attention habits. Currently, there are two main approaches to visual attention: one is to directly learn a weight distribution for image features and then apply this weight distribution to the original features, providing different feature responses for subsequent tasks such as image classification and recognition. The core idea is to identify correlations within the original image data to highlight important features, thereby improving recognition efficiency. Key approaches include pixel attention and multi-order attention. The other is to spatially transform the data or data features (feature maps) to focus on a specific part of the image (region of interest). The core idea is to find the spatial transformation relationship between the region of interest and the original image, use network learning to identify key features, and "crop" the region of interest, thereby highlighting the image region of interest and improving recognition efficiency. Key approaches include the Spatial Transformer Network (STN). Both approaches enable the task to focus on important features and ignore unimportant ones, thereby improving task efficiency.

[0033] In medical images, lesions are identified not only by size or edge features, but also by density differences between the region and surrounding areas. Therefore, medical diagnosis places high demands on image quality, and subtle structural changes in the image are crucial. However, the STN algorithm faces two major challenges when applied to medical image processing: 1) Affine transformations can cause morphological distortion of the lesion area; 2) Interpolation algorithms can lead to loss of high-frequency components in scaled images, blurring image edges to a certain extent. Therefore, when using the STN algorithm for medical image processing, a significant amount of subtle structural information in the original image is lost, resulting in an inaccurate display of tissue structure and its adjacent relationships, ultimately obscuring or missing lesions.

[0034] Take a lung nodule in a CT image as an example. Figure 1-a As shown, Figure 1-a Figure 4 shows an image of a small lung nodule obtained by CT scan. Figure 1-b As shown, Figure 1-b The STN algorithm according to the prior art is shown. Figure 1-aThe image of the small lung nodule obtained by image processing has a certain degree of distortion due to the addition of deformation factors in the image transformation; Figure 1-c As shown, Figure 1-c The interpolation algorithm according to the prior art is shown. Figure 1-a The image of the small lung nodule obtained by image processing becomes blurred to a certain extent due to interpolation.

[0035] The concept of this application is to improve two problems existing in the STN attention mechanism in medical image diagnosis, and form a medical image target identification model. When diagnosing medical images, the medical image target identification model can not only make full use of the characteristics of the spatial transformation network's attention to the lesion area, but also overcome the image quality loss caused by the extraction process of the lesion area, thereby improving the performance of deep learning in medical diagnosis.

[0036] Figure 2 A schematic flow chart of a medical image target identification method according to an embodiment of the present application is shown. The method is implemented by a medical image target identification model, which includes a spatial transformation network, a feature extraction network, and a classification prediction network connected in sequence; Figure 2 It can be seen that this application includes at least steps S210 to S240:

[0037] Step S210: Acquire a pair of images to be predicted, wherein the pair of images to be predicted includes a corresponding first image and a second image, wherein the lesion in the first image is marked.

[0038] The medical image target identification method of the present application can be implemented through a medical image target identification model. Figure 3 A schematic diagram of the structure of a medical image target identification model according to an embodiment of the present application is shown. Figure 3 As can be seen, the medical image object identification model 300 includes a spatial transformation network 310, a feature extraction network 320, and a classification prediction network 330, which are connected in sequence. The spatial transformation network 310 serves as the input of the medical image object identification model 300, and the classification prediction network 330 serves as the output of the medical image object identification model 300. Taking medical CT images as an example, after obtaining a patient's CT image, the same image is copied to obtain an original image and a copy. Both images are identical before annotation. One of the images is recorded as the first image, and the other is recorded as the second image.

[0039] One of the images is annotated. Specifically, the outline of the lesion (e.g., a lung nodule) in the image is circled. This annotation can be done manually, such as by having a medical expert outline it, or automatically, such as by placing an automatic outlining tool in front of the spatial transformation network 310 of the medical image target identification model 300. This tool outlines the outline of the identified nodule and then sends it to the medical image target identification model 300 for identification as benign or malignant. After outlining, the remaining parts of the outlined image are processed and can be set to black (the pixel value or depth value of the pixel is set to a preset value).

[0040] It can also be understood that the first image is an image that only contains nodule information, and the second image is an image that contains both nodule and surrounding nodule information (other normal tissues).

[0041] Step S220: Based on the spatial transformation network, determine Euclidean transformation parameters based on the first image; perform spatial transformation and interpolation processing on the first image based on the Euclidean transformation parameters to obtain a first region of interest; and perform spatial transformation and cropping processing on the second image based on the Euclidean transformation parameters to obtain a second region of interest.

[0042] In existing technology, STN algorithms, when applied to medical imaging, typically use affine transformations, projective transformations, and other techniques to process images. However, CNNs in deep learning use pooling layers to ensure spatial invariance. However, these layers are limited by pre-selected fixed-size pooling kernels (response fields are fixed and localized), resulting in insufficient spatial invariance. STNs learn the parameters of spatial transformations to restore deformed objects to their normal positions, thereby enhancing spatial invariance. In reality, lung nodules undergo some deformation due to local pressure changes caused by body position and breathing, but the deformation is extremely small, and they do not experience significant elastic deformation due to uneven forces. Therefore, transformed images generated by affine and projective transformations are almost nonexistent in the real world. Furthermore, unlike natural images, the circularity of nodules is considered an important indicator for distinguishing benign from malignant nodules by experienced physicians. Therefore, deformation is a characteristic that hinders diagnosis. Applying too many large-scale transformations is not only meaningless but also weakens features such as circularity.

[0043] The inventors discovered that the Euclidean transformation preserves the length and angle of vectors. Therefore, under this transformation, the length, area, and shape of a nodule remain unchanged. This transformation provides spatial transformation capabilities, overcoming nodule pixel rotation caused by positional errors during CT imaging. Therefore, the Euclidean transformation enables the network to better infer nodule features and derive nodule posture from texture and shape.

[0044] Therefore, this application only uses the Euclidean transformation to reflect the position transformation of the nodule in the image, and does not use other transformations. The Euclidean transformation can be expressed in the matrix form of the following formula 1:

[0045]

[0046] Where R is the rotation matrix and t is the translation vector.

[0047] Specifically, the first image and the second image are divided into two subtasks and processed separately. For the first image, after it is input into the spatial transformation network, the spatial transformation network learns the first image to generate appropriate Euclidean transformation parameters. In some embodiments of the present application, the spatial transformation network can be constructed based on the spatial transformation attention mechanism, but it should be noted that the spatial transformation network of the present application only supports Euclidean transformation and does not support other spatial transformations such as radial transformation.

[0048] More specifically, in some embodiments of the present application, the spatial transformation network is provided with a Euclidean parameter learning subnetwork; determining the Euclidean transformation parameters based on the first image includes: inputting the first image into the spatial transformation network, and enabling the Euclidean parameter learning subnetwork to learn the position information of the marked area in the first image to obtain the Euclidean transformation parameters, and the Euclidean transformation parameters include: rotation parameters, first translation parameters and second translation parameters.

[0049] That is to say, the Euclidean parameter learning subnetwork learns the first image by using the Euclidean transformation parameter, denoted as θ. The Euclidean transformation parameter θ(R, tx, ty) has three degrees of freedom, where R represents the degree of rotation, and tx and ty both represent the amount of translation.

[0050] Then the learned Euclidean transformation parameters are used to process the first image and the second image respectively.

[0051] For the first image, spatial transformation and interpolation processing are performed on the first image according to the learned Euclidean transformation parameters. Specifically, in some embodiments of the present application, spatial transformation and interpolation processing are performed on the first image according to the Euclidean transformation parameters to obtain a first region of interest, including: performing an equidistant transformation on each pixel point in the first image based on the Euclidean transformation parameters to obtain first position information of the first region of interest in the first image; and interpolation processing is performed on each pixel point in the first region of interest according to the first position information to obtain the first region of interest.

[0052] The pixel values ​​(or depth values) of each pixel in the first image are multiplied by the matrix formed by the Euclidean transformation parameters, thereby achieving an isometric transformation of the first image. The first image only contains nodule information. Through the Euclidean transformation, the nodule information is enhanced in the first image. The Euclidean transformation of the target coordinate point and the original coordinate point is performed. The coordinate position of the region of interest in the original image is obtained through this transformation, that is, the coordinate position of the first region of interest (nodule part) in the first image is obtained through this transformation, which is recorded as the first position information. Based on the first position information, the first region of interest can be separated from the first image. Then, each pixel point of the separated first region of interest is interpolated to obtain the first region of interest. The target coordinate point and the original coordinate point are transformed and interpolated according to the predicted Euclidean transformation parameters, and an image of the region of interest without any information loss can be obtained, thereby avoiding the loss of detail information.

[0053] The second image can be cropped according to the Euclidean transformation parameters. Specifically, in some embodiments of the present application, the second image is spatially transformed and cropped according to the Euclidean transformation parameters to obtain the second region of interest, including: performing an equidistant transformation on each pixel point in the second image based on the Euclidean transformation parameters to obtain second position information of the second region of interest in the second image; and cropping the second region of interest from the second image according to the second position information.

[0054] The pixel value (or depth value) of each pixel point in the second image is multiplied by the matrix formed by the Euclidean transformation parameters, thereby realizing the isometric transformation of the second image. Through the Euclidean transformation, the nodule information is enhanced in the second image, and the Euclidean transformation of the target coordinate point and the original coordinate point is performed. The coordinate position of the region of interest in the original image is obtained through this transformation, that is, the coordinate position of the second region of interest (nodule part) in the second image is obtained through this transformation, which is recorded as the second position information here. According to the second position information, the second region of interest can be separated from the second image, that is, cropped out, to select the most relevant nodule area in the image, such as cropping the second image from an m×m image to an n×n image (n<m).

[0055] In addition, in some embodiments of the present application, the method further includes performing spatial transformation and cropping on the second image according to the Euclidean transformation parameters to obtain a second region of interest, and further includes normalizing the pixel values ​​of the second region of interest, and using the resulting image as the final second region of interest. Normalizing the scale of the region of interest through normalization can reduce subsequent classification load and achieve better classification performance.

[0056] Step S230: Based on the feature extraction network, feature extraction is performed on the first region of interest and the second region of interest respectively to obtain a first feature and a second feature.

[0057] After obtaining the first region of interest and the second region of interest, the first region of interest and the second region of interest are respectively input into a feature extraction network, which is responsible for extracting regional features of the regions of interest.

[0058] In some embodiments of the present application, the feature extraction network can be formed based on a convolutional neural network (CNN). In the CNN feature extraction network, the two most critical parameters of the network can be set according to requirements: the number of layers and the kernel size, such as according to computing power and identification accuracy.

[0059] The feature extraction network extracts features from the first region of interest and the second region of interest, and obtains corresponding first features and second features, respectively. When extracting features from the first region of interest and the second region of interest, the same feature extraction network can be called to process the first region of interest and the second region of interest, respectively. Alternatively, two feature extraction subnetworks can be set in the feature extraction network, with the first feature extraction subnetwork being used to extract features from the first region of interest and the second feature extraction subnetwork being used to extract features from the second region of interest. The first feature extraction subnetwork and the second feature extraction subnetwork can be the same or different and can be set as needed; however, the main functions of the first feature extraction subnetwork and the second feature extraction subnetwork are both to extract features from the first region of interest and the second region of interest.

[0060] Step S240: Based on the classification prediction network, the first feature and the second feature are fused to obtain a fused feature, and the fused feature is classified to obtain a target identification result.

[0061] After obtaining the first feature corresponding to the first image and the second feature corresponding to the second image, the first feature and the second feature are fused using a feature fusion method to obtain a fused feature. The first feature and the second feature are usually presented in the form of vectors or matrices. In some embodiments of the present application, in the above method, the fusion of the first feature and the second feature to obtain the fused feature includes: if the first feature and the second feature are one-dimensional vectors, then connecting each element of the second feature to the last element of the first feature in the original order to obtain a fused vector;

[0062] If the first feature and the second feature are matrices, then according to the specified splicing form, each element of the second feature is placed in the corresponding position after the first feature in the original order to obtain a fusion vector.

[0063] For example, assuming that the first feature is a vector (a, b, c) and the second feature is a vector (d, e), when splicing, the elements in the second feature are "placed" after the last element of the first feature according to their original order. The fused vector is recorded as vector z, and the fused vector z is (a, b, c, d, e).

[0064] In some other embodiments of the present application, if the first feature and the second feature are matrices; fusing the first feature and the second feature to obtain a fusion vector includes: placing each element of the second feature in the original order according to a specified splicing form to the corresponding position after the first feature to obtain a fusion vector.

[0065] Assume that the first feature and the second feature are isomorphic matrices, and assume that the first feature and the second feature are both 2*2 two-dimensional matrices, respectively expressed as: and Specify vertical splicing. When splicing, follow the original order of the elements in the second feature and "place" them after the last row of the first feature. The fusion vector is recorded as vector z. The fusion vector z is a 2*4 matrix, specifically:

[0066]

[0067] If the first and second features are heterogeneous matrices, simply set the missing elements to empty.

[0068] The fused features are then classified and predicted. In some embodiments of the present application, a classification prediction network can be constructed using a fully connected layer and a loss function. Here, the classification prediction network is usually a binary classification model. For example, to determine whether a nodule is benign or malignant, benign and malignant are used as two categories of the classification prediction network. By inputting the fused features into it, the benign or malignant nature of the lesion can be obtained.

[0069] Depend on Figure 2It can be seen from the method shown that the present application has designed a medical image target identification model, which includes a spatial transformation network, a feature extraction network and a classification prediction network connected in sequence; when the image to be predicted is predicted and classified, images of the same part of the same patient are stored separately, one of which has the lesion manually or automatically outlined and marked in advance, and then the first image and the second image are respectively input into the medical image target identification model, and the two first enter the spatial transformation network, and the Euclidean transformation parameters of the first image are learned through the spatial transformation network, and the first image is spatially transformed and interpolated according to the learned Euclidean transformation parameters to obtain a first region of interest in which the lesion in the first image is enhanced, thereby achieving The region of interest in the first image is restored by linear interpolation, and the restored image is used for back propagation learning, so that the lesion features in the first image are enhanced; for the second image, it is cropped according to the Euclidean transformation parameters learned from the first image to obtain a second region of interest with a suitable position and no information loss; then the first region of interest and the second region of interest are sent to the feature extraction network, and their information is extracted respectively to obtain the first feature and the second feature respectively; finally, the first feature and the second feature are sent to the classification prediction network, and the first feature and the second feature are fused by feature fusion to obtain the fused feature, and the fused features are classified to obtain the target identification result.

[0070] This application addresses the prior art, in which the attention mechanism based on spatial transformation uses an interpolation method to generate a region of interest image to ensure the continuity of the network's reverse gradient transfer. This operation will result in loss of detail in the generated image and may cause loss of information of some key diagnostic features, thereby leading to missed diagnosis or misdiagnosis. This application proposes a mechanism for restoring the original image with the addition of a region of interest. Two images of the same affected area are cleverly used, one of which is annotated and the Euclidean transformation parameters are learned. The region of interest in the first image is restored using linear interpolation. The restored image is used for backpropagation learning, so that the lesion features in the first image are enhanced, and the Euclidean transformation parameters are learned. The learned Euclidean transformation parameters are applied to the cropping of the other image to obtain a region of interest without any information loss, thereby obtaining multiple features of the target lesion without loss, thereby ensuring that the image detail information in the original image is not lost, and significantly improving the accuracy of target identification in medical images. In addition, this application divides the overall task into two subtask branches, redistributing the network's learning ability, thereby reducing the difficulty of the original task and making the network model easier to train.

[0071] Figure 3 A schematic diagram showing the flow of data in a medical image target identification model according to an embodiment of the present application is shown. Figure 3It can be seen that the medical image target identification model in this embodiment includes an automatic labeling network, a spatial transformation network, a feature extraction network and a classification prediction network. The spatial transformation network 310 is provided with a Euclidean parameter learning subnetwork and a spatial transformation subnetwork, the feature extraction network is provided with a first feature extraction subnetwork and a second feature extraction subnetwork, and the classification prediction network is provided with a feature fusion subnetwork and a fully connected layer.

[0072] A CT image of the patient is acquired, and the CT image is copied to obtain a first image and a second image.

[0073] The first image is fed into an automatic annotation network for outlining to obtain an annotated first image.

[0074] The labeled first image and the second image enter the spatial transformation network respectively. Specifically, the first image enters the Euclidean parameter learning subnetwork, and the Euclidean parameter learning subnetwork learns the first image to obtain Euclidean transformation parameters.

[0075] Then, the spatial transformation subnetwork is entered to perform spatial transformation and interpolation on the first image based on the Euclidean transformation parameters to obtain the first region of interest.

[0076] The second image directly enters the spatial transformation subnetwork, and the second image is spatially transformed and cropped based on the Euclidean transformation parameters to obtain the second region of interest.

[0077] The first region of interest enters the first feature extraction subnetwork, and the second region of interest enters the second feature extraction subnetwork, and feature extraction is performed respectively to obtain the first feature and the second feature.

[0078] The first feature and the second feature enter the feature fusion sub-network for feature fusion to obtain a fused feature.

[0079] The fused features enter the fully connected layer and output the target classification results.

[0080] In addition, the present application is compared with the reference documents to verify the results of the present application, which are briefly described below:

[0081] The Lung Image Database Consortium (LIDC-IDRI) database is the world's largest public resource for lung nodules. It contains 1,012 cases, including chest CT scans and associated XML files. Each XML file records the annotations of four experienced chest radiologists, including location, malignancy assessment, and grades labeled as "1," "2," "3," "4," and "5," as well as eight other characteristics of detected nodules ≥3 mm.

[0082] Based on the principle of slice thickness <2.5mm, this application extracted 888 CT scan images. The matrix size of the CT scan is 512×512 pixels, the axial resolution ranges from 0.46×0.46mm to 0.98×0.98mm, and the most typical resolution of thin-slice CT scans is 0.78mm. Nodules with a diameter of less than 3mm are considered irrelevant to lung cancer CT scans and are therefore excluded from the learning set. Each nodule is rated on a five-level scale, with nodules marked "1" and "2" being benign and nodules marked "4" and "5" being malignant. The label "3" indicates that the corresponding nodule is uncertain as to whether it is benign or malignant. A total of 1,372 nodules were finally obtained, including 379 benign nodules, 573 undetermined nodules, and 420 malignant nodules, with nodule diameters ranging from 3.0mm to 37.32mm. In some studies, nodules labeled "3" can be divided into two different treatments: (1) they are divided into benign nodules labeled "1" and "2"; (2) they are divided into benign nodules labeled "4" and "5". In the experiment of this application, the application abandoned the nodules labeled "3" and obtained a total of 4622 nodule slices in the LIDC-IDRI and hospital datasets, including 1597 benign slices and 3025 malignant slices.

[0083] In the experiments of this application, PyTorch was used to build the proposed network. These experiments were conducted on a computer with an AMD Ryzen 1500X CPU operating at 3.5GHz, 24GB of RAM, and an NVIDIA RTX3080Ti 12GB GPU. A 5-fold cross-validation method was used to create training, validation, and test data. All data were randomly divided into 5 subsets, 4 of which were used for training and 1 for validation and testing. In order to compare the performance of the proposed model for object identification through medical imaging (herein denoted as PLADL) with other deep learning networks, this application conducted relevant experiments to demonstrate the superiority of the proposed PLADL network. One experiment was a ResNet50 classification method based on the nodule segmentation network, which is a general method for overcoming interference information; the other experiment was an STN classification experiment that showed advanced performance in feature extraction. For objective comparison, these experiments used the same training, validation, and test images, all of which were nodule images on CT slices. In addition, for fair evaluation, all networks had the same post-processing method. The experimental results are shown in Table 1 below. Experimental results show that PLADL has superior performance in classifying benign and malignant nodules.

[0084] Table 1

[0085]

[0086] Furthermore, our proposed medical image object identification model (PLADL) was compared with several recently proposed state-of-the-art networks, including Res-trans, MK-SSAC, MIL, Shaffie et al., and Fuse-TSD. The comparison results are shown in Table 2. The Shaffie et al. model achieved higher AUC and accuracy. It is worth noting that its input data is three-dimensional. Previous studies have shown that networks based on three-dimensional patches achieve better classification performance, but at the cost of higher model complexity and computational complexity.

[0087] Table 2

[0088]

[0089] Figure 5 A schematic diagram of the structure of a medical image target identification device according to an embodiment of the present application is shown. The device deploys a medical image target identification model, which includes a spatial transformation network, a feature extraction network, and a classification prediction network connected in sequence. The medical image target identification device 500 includes:

[0090] An acquisition unit 510 is configured to acquire an image pair to be predicted, the image pair to be predicted comprising a corresponding first image and a second image, wherein a lesion in the first image is labeled;

[0091] a positioning unit 520 configured to determine Euclidean transformation parameters based on the spatial transformation network and the first image; perform spatial transformation and interpolation processing on the first image based on the Euclidean transformation parameters to obtain a first region of interest; and perform spatial transformation and cropping processing on the second image based on the Euclidean transformation parameters to obtain a second region of interest;

[0092] An extraction unit 530 is configured to extract features from the first region of interest and the second region of interest based on a feature extraction network to obtain a first feature and a second feature;

[0093] The classification unit 540 is configured to fuse the first feature and the second feature based on the classification prediction network to obtain a fused feature, and classify the fused feature to obtain a target identification result.

[0094] In some embodiments of the present application, in the above-mentioned device, the spatial transformation network is provided with a Euclidean parameter learning subnetwork; the positioning unit 520 is used to input the first image into the spatial transformation network, and enable the Euclidean parameter learning subnetwork to learn the position information of the marked area in the first image to obtain the Euclidean transformation parameters, and the Euclidean transformation parameters include: rotation parameters, first translation parameters and second translation parameters.

[0095] In some embodiments of the present application, in the above-mentioned device, the positioning unit 520 is used to perform an equidistant transformation on each pixel point in the first image based on the Euclidean transformation parameters to obtain first position information of the first region of interest in the first image; and interpolate each pixel point of the first region of interest according to the first position information to obtain the first region of interest.

[0096] In some embodiments of the present application, in the above-mentioned device, the positioning unit 520 is used to perform an equidistant transformation on each pixel point in the second image based on the Euclidean transformation parameters to obtain second position information of the second region of interest in the second image; and based on the second position information, the second region of interest is cropped out from the second image.

[0097] In some embodiments of the present application, in the above-mentioned apparatus, the positioning unit 520 is further configured to perform normalization processing on each pixel value of the second region of interest, and use the processed image as the final second region of interest.

[0098] In some embodiments of the present application, in the above-mentioned device, the classification unit 540 is used to, if the first feature and the second feature are one-dimensional vectors, connect each element of the second feature in the original order to the last element of the first feature to obtain a fusion vector; if the first feature and the second feature are matrices, then according to a specified splicing form, place each element of the second feature in the original order to the corresponding position after the first feature to obtain a fusion vector.

[0099] In some embodiments of the present application, in the above-mentioned device, the spatial transformation network is constructed based on the spatial transformation attention mechanism; the feature extraction network is constructed based on the convolutional neural network; and the classification prediction network is constructed based on the fully connected layer and the loss function.

[0100] It should be noted that the above-mentioned medical image target identification device can implement the above-mentioned medical image target identification method one by one, and will not be described in detail.

[0101] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 6At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0102] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0103] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0104] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a medical image target identification device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the aforementioned method.

[0105] The above application Figure 5The method performed by the medical image target identification device disclosed in the illustrated embodiment can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0106] The electronic device may also perform Figure 5 Method for executing the medical image target identification device, and realizing the medical image target identification device in Figure 5 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0107] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 5 The method executed by the medical image target identification device in the illustrated embodiment is specifically used to execute the aforementioned method.

[0108] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0109] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0110] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0112] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0113] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0114] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for identifying medical image targets, characterized in that: The method is implemented by a medical image target identification model, which includes a spatial transformation network, a feature extraction network, and a classification prediction network connected in sequence; the method includes: Acquire a pair of images to be predicted, the pair comprising a corresponding first image and a second image, wherein a lesion in the first image is labeled; Determining Euclidean transformation parameters based on the first image based on the spatial transformation network; performing spatial transformation and interpolation processing on the first image based on the Euclidean transformation parameters to obtain a first region of interest; and performing spatial transformation and cropping processing on the second image based on the Euclidean transformation parameters to obtain a second region of interest; Based on a feature extraction network, feature extraction is performed on the first region of interest and the second region of interest respectively to obtain a first feature and a second feature; Based on the classification prediction network, the first feature and the second feature are fused to obtain a fused feature, and the fused feature is classified to obtain a target identification result.

2. The method according to claim 1, characterized in that The spatial transformation network is provided with a Euclidean parameter learning subnetwork; and determining the Euclidean transformation parameters according to the first image includes: The first image is input into the spatial transformation network, and the Euclidean parameter learning subnetwork is enabled to learn the position information of the marked area in the first image to obtain the Euclidean transformation parameters, where the Euclidean transformation parameters include: a rotation parameter, a first translation parameter, and a second translation parameter.

3. The method according to claim 2, characterized in that The performing spatial transformation and interpolation processing on the first image according to the Euclidean transformation parameters to obtain a first region of interest includes: Performing an equidistant transformation on each pixel in the first image based on the Euclidean transformation parameters to obtain first position information of the first region of interest in the first image; According to the first position information, interpolation processing is performed on each pixel point of the first region of interest to obtain the first region of interest.

4. The method according to claim 2, characterized in that The performing spatial transformation and cropping processing on the second image according to the Euclidean transformation parameters to obtain a second region of interest includes: Performing an equidistant transformation on each pixel in the second image based on the Euclidean transformation parameters to obtain second position information of the second region of interest in the second image; The second region of interest is cropped from the second image according to the second position information.

5. The method according to claim 4, characterized in that The performing spatial transformation and cropping processing on the second image according to the Euclidean transformation parameters to obtain a second region of interest further includes: Normalization processing is performed on the pixel values ​​of the second region of interest, and the processed image is used as the final second region of interest.

6. The method according to claim 1, characterized in that The fusing the first feature and the second feature to obtain a fused feature includes: If the first feature and the second feature are one-dimensional vectors, then concatenate each element of the second feature to the last element of the first feature in their original order to obtain a fused vector; If the first feature and the second feature are matrices, then according to the specified splicing form, each element of the second feature is placed in the corresponding position after the first feature in the original order to obtain a fusion vector.

7. The method according to any one of claims 1 to 6, characterized in that The spatial transformation network is constructed based on the spatial transformation attention mechanism; the feature extraction network is constructed based on the convolutional neural network; and the classification prediction network is constructed based on the fully connected layer and the loss function.

8. A medical image target identification device, characterized in that: The device is deployed with a medical image target identification model, which includes a spatial transformation network, a feature extraction network, and a classification prediction network connected in sequence; the device includes: an acquisition unit, configured to acquire a pair of images to be predicted, the pair of images to be predicted comprising a corresponding first image and a second image, wherein a lesion in the first image is marked; a positioning unit, configured to determine Euclidean transformation parameters based on the spatial transformation network and the first image; perform spatial transformation and interpolation processing on the first image according to the Euclidean transformation parameters to obtain a first region of interest; and perform spatial transformation and cropping processing on the second image according to the Euclidean transformation parameters to obtain a second region of interest; an extraction unit, configured to perform feature extraction on the first region of interest and the second region of interest respectively based on a feature extraction network to obtain a first feature and a second feature; A classification unit is used to fuse the first feature and the second feature based on the classification prediction network to obtain a fused feature, and classify the fused feature to obtain a target identification result.

9. An electronic device comprising: processor; as well as A memory arranged to store computer-executable instructions, wherein when the executable instructions are executed, the processor executes the medical image target identification method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device executes the medical image object identification method according to any one of claims 1 to 7.

Images