A pathological image multi-channel fusion automatic matching method and system

Abstract

This invention discloses a multi-channel fusion automatic matching method and system for pathological images, used in the field of medical image matching to overcome the technical challenge of high-precision matching of pathological images, and to meet the clinical needs of cancer auxiliary diagnosis by locating local common regions in pathological images. First, multiple single-modal image matching channels are created by normalizing image modalities. Then, image matching is performed in each channel, including extracting feature points, constructing feature vectors, and determining matching features. Next, based on the consistency of the mapping relationship of the matching features, the multi-channel matching features are fused to determine the matching result. Finally, the highly consistent matching result is used to assist in the precise clinical location of local common regions in pathological images. The pathological image matching method and apparatus provided by this invention avoids the interference of modal differences in pathological image matching by normalizing image modalities, and enhances the consistency of matching results by fusing multi-channel matching features, avoiding erroneous matching and improving matching accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision research, specifically relating to a method and system for automatic matching of multi-channel fusion of pathological images. Background Technology

[0002] In the field of computer vision research, particularly in medical image processing, the analysis and diagnosis of pathological images are crucial for the early detection and treatment of diseases such as cancer. However, existing pathological image processing methods face a series of technical challenges when dealing with the complex joint analysis of multiple pathological images.

[0003] When making a pathological diagnosis, pathologists need to combine biomarker features from multiple pathological images to assess tumor-related characteristics. However, differences in physician experience and background lead to variations in the ability to locate similar biomarkers in different images, affecting diagnostic accuracy and consistency.

[0004] In the field of computer image registration, modal differences in pathological images have become a limiting factor for high-precision matching. Images obtained under different devices and conditions vary significantly in color, brightness, and other aspects. Traditional multimodal matching struggles to accurately predict the transformation matrix, resulting in local common regions containing a large number of irrelevant markers, which affects the reliability of clinical diagnosis.

[0005] Furthermore, the scarcity of medical equipment limits clinical diagnosis. When different pathologists diagnose the same patient multiple times, they need to frequently locate common areas in pathological images; however, subjective factors influence manual selection, leading to low efficiency and difficulty in reuse. Therefore, there is an urgent need for pathological image matching devices to efficiently reuse common areas in pathological images and meet the needs of cancer diagnosis. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention discloses a method and system for automatic matching of multi-channel fusion of pathological images.

[0007] To achieve the above objectives, the present invention provides a method and system for automatic matching of multi-channel fusion of pathological images, comprising the following steps:

[0008] Step 1: Acquire pathological images in multiple modalities. Using the pathological images in each modality as a reference, normalize the pathological images in the remaining modalities to the corresponding modalities to obtain multiple images in each modality.

[0009] Step 2: Perform image matching on the images of each modality in turn to obtain the matching features for each modality;

[0010] Step 3: Combine the matching features under each modality, and fuse the matching features under multiple modalities to obtain the fused matching features;

[0011] Step 4: Based on the fused matching features, assist in the precise localization of common regions in pathological images;

[0012] Preferably, the above-mentioned method and system for automatic matching of multi-channel fusion of pathological images is characterized in that step 1 includes:

[0013] Step 1.1: Normalize image modalities to construct matching channels;

[0014] Modality conversion algorithms, including but not limited to color space transformation algorithms, nonlinear mapping algorithms, and cycle consistency generative adversarial algorithms, are used to convert images of different modalities to a consistent modality. The characteristic of this algorithm is that the purpose of normalizing the image modality is to construct matching channels.

[0015] Step 1.2: Each matching channel contains a pair of unmatched image GLs, one being the original pathological image without modality conversion, and the other being generated by the image modality conversion network.

[0016] Step 1.3: The image GL in each matching channel has a single modality, that is, the image G or L generated by the image modality conversion network has the same image modality as the original pathological image L or G.

[0017] Step 1.4: Perform image GL matching within the same matching channel; image GL matching is not required in different matching channels.

[0018] Step 1.5: Use the same or different matching algorithms in different matching channels, including but not limited to SIFT and SURF algorithms. For example, use the SIFT algorithm to match image GL in matching channel 1, and use the SIFT or SURF algorithm to match image GL in matching channel 2.

[0019] Preferably, step 2 includes:

[0020] Step 2.1: Use a matching algorithm to match the image GL across multiple matching channels, with each matching channel generating a set of matching features.

[0021] Step 2.2: The matching features under each matching channel are composed of matching feature point pairs (X... p ,Y q M p N q ) indicates that (X) p ,Y q ) and (M p N q ) are the pixel coordinates of image G and image L, respectively. p,q are the pixel dimensions of image GL.

[0022] Preferably, step 3 includes:

[0023] Step 3.1: The mapping relationship of image GL is represented by rigid transformation matrix and non-rigid transformation matrix, wherein the rigid transformation matrix and non-rigid transformation matrix are obtained by prediction based on multi-matching channel matching features;

[0024] It can be obtained by calling the rigid transformation matrix and non-rigid transformation matrix functions of the python-cv2 library;

[0025] The matching features of each matching channel can generate a rigid transformation matrix and a non-rigid transformation matrix.

[0026] Step 3.2: The consistency of the mapping relationship of the image GL is determined by the angular error of the relative angle of the image GL between different matching channels and the similarity of the common region of the image GL in the same matching channel.

[0027] Step 3.3: When there is only one matching channel for a pathological image, the matching feature is directly used as the matching result.

[0028] Step 3.4: When there are multiple pathological image matching channels, in order to achieve the purpose of fusing matching features from multiple matching channels, calculate the angle error between the relative angle of image GL in each matching channel and the relative angle of image GL in all matching channels, and statistically analyze the histogram distribution of angle errors less than threshold T1. Calculate the similarity level of image GL within the matching channel corresponding to the most frequent histogram, and fuse the matching features with similarity levels less than threshold T2. The result is the matching result.

[0029] The consistency of the mapping relationship described in step 3 is characterized by including:

[0030] The rigid transformation matrix is ​​used to calculate the relative angle of the image GL. The relative angle represents the angular error of the image GL.

[0031] The angle error is less than the threshold T1. The smaller T1 is, the smaller the angle error, which indicates that the consistency of the matching features of the image GL between different matching channels is better.

[0032] The non-rigid transformation matrix represents the mapping relationship of image GL, and therefore can be used to locate common regions of image GL. Specifically, the localization process involves multiplying the center point coordinates (x, y) of the region of interest with the non-rigid transformation D to obtain the mapped coordinates (x', y'), which are the mapped coordinates to other corresponding pathological images. Thus, (x, y) and (x', y') can locate the common region. The common region is used to calculate the similarity of image GL in the same matching channel.

[0033] The similarity is determined by image similarity metrics, including but not limited to the use of structural similarity index or mean squared error. The similarity of common regions is less than a threshold T2, and the smaller T2 is, the higher the GL similarity of images in the same matching channel.

[0034] The angle error required for fusing multi-matching channel matching features in step 3 is as follows:

[0035] The rigid transformation matrix predicted based on multi-match channel matching features is as follows:

[0036]

[0037] Among them, R n This represents the rigid transformation matrix. n represents the number of matching channels. r 11 r 12 r 13 , representing the x-axis direction information of the image rotation. 21 r 22 r 23 This indicates the y-axis direction information for image rotation. 31 r 32 r 33 This represents the z-axis direction information of the image rotation. The relative angle θ of the image GL in each matching channel of the rotation matrix is ​​given by the following formula:

[0038]

[0039]

[0040]

[0041] θ=[θ x ;θ y ;θ z ]

[0042] Where, θ x ,θ y ,θ z These represent the rotation angles of the pathological image around the X, Y, and Z axes, respectively. The formula for calculating the angle error is as follows:

[0043]

[0044] Among them, E i This represents the angular error of the relative angle of image GL in the i-th matching channel. i∈[1,2,3,…,n] represents the order of the matching channels. For ease of description, it is defined as follows:

[0045]

[0046] in This represents the angular error in the X, Y, Z axes between the relative angle of image GL in the i-th matching channel and the relative angle of image GL in the n-th matching channel.

[0047] Step 3, which involves fusing the similarity required for multi-channel matching features, is as follows:

[0048] Calculate the data H for plotting the angle error histogram.

[0049] H i =sum(E i <T1)

[0050] Among them, E i T1 represents the angular error of the relative angle of image GL in the i-th matching channel. T1 is the angular error threshold. i∈[1,2,3,…,n] represents the order of the matching channels.

[0051] Determine the order j of the matching channels corresponding to the most frequent histogram.

[0052] H i＝j =max(H)

[0053] Where H represents the data from the angle error histogram. j represents the most frequent matching channel. i∈[1,2,3,…,n] indicates the order of the matching channels.

[0054] The matching channel that has the same mapping relationship with the j-th matching channel is determined based on the matching characteristics of the j-th matching channel.

[0055] U = E j <T1

[0056] in, Therefore, U represents the index of all matching channels with consistent mapping relationships. E j T1 represents the angular error of the relative angle of image GL in the most frequently matched channel j. T1 is the angular error threshold. This represents the angular error in the X, Y, Z axes between the relative angle of image GL in the most frequent matching channel j and the relative angle of image GL in the nth matching channel.

[0057] Transform image L using a non-rigid transformation matrix to generate image W that shares a common region with image G.

[0058] W i =V i L i ,if U i =1

[0059] Where W is the image W. V is the non-rigid transformation matrix. L is the image L. i∈[1,2,3,…,n] represents the order of the matching channels.

[0060] Calculate the similarity level between image G and image W.

[0061]

[0062] Where S represents the similarity level of images GL in matching channels with consistent mapping relationships. f is a metric for measuring image similarity, including but not limited to using structural similarity index or mean squared error. G is image G.

[0063] The fusion of multi-matching channel matching features described in step 3 further includes the following sub-steps:

[0064] The matching results are obtained by fusing matching features from multiple matching channels.

[0065] K = S <T2

[0066]

[0067] Where K represents the matching channel index where the similarity level of image GL is less than T2. ​​S represents the similarity level of image GL in matching channels with consistent mapping relationships. T2 is the similarity threshold of the common region. F is the matching result fused with matching features from multiple matching channels.

[0068] Preferably, step 4 includes:

[0069] Step 4.1: The user uses the mouse to select a region of interest on the pathological image to determine the coordinates of the center point of the region of interest.

[0070] Step 4.2: Based on the matching results obtained in Step 3, calculate the non-rigid transformation matrix D. Specifically, this can be obtained by calling the non-rigid transformation matrix function in the python-cv2 library.

[0071] Step 4.3: Calculate the mapped coordinates of the center point of the region of interest in other corresponding pathological images based on the non-rigid transformation matrix D. Specifically, the calculation process involves multiplying the center point coordinates (x, y) of the region of interest by the non-rigid transformation D to obtain new coordinates (x', y'), which are the mapped coordinates to other corresponding pathological images.

[0072] Step 4.4: Using the mapped coordinates as the center point of the region of interest, extract the region of interest and combine it with the region of interest selected by the user to form a common area of ​​the pathological image.

[0073] A system for automatic matching of multi-channel fusion of pathological images, characterized in that it includes:

[0074] Matching channel unit, storage unit, rigid transformation matrix determination unit, angle error determination unit, non-rigid transformation matrix V determination unit, similarity determination unit, fusion unit, mouse unit, and non-rigid transformation matrix D determination unit;

[0075] The matching channel unit is used to convert the original pathological image modality and construct a pathological image matching channel. The converted image and the original pathological image (with the same modality) are used to form a single-modality matching channel.

[0076] The storage unit is used to store the matching features of the pathological image matching channel.

[0077] The rigid transformation matrix determination unit is used to calculate the rigid transformation matrix obtained from the matching features in each matching channel.

[0078] The angle error determination unit is used to determine the relative angle error between different pathological images. It calculates the relative angle of the pathological images in each matching channel, compares the errors between the relative angles in different matching channels, and uses a threshold to determine the matching features that meet the angle error requirements.

[0079] The non-rigid transformation matrix V determination unit is used to calculate the non-rigid transformation matrix V obtained by calculating the matching features within the matching channel that meet the angle error requirements. The non-rigid transformation matrix V is calculated using matching features that meet the angle error and are defined by the angle error determination unit, and is stored in the storage unit.

[0080] The similarity determination unit transforms the pathological image according to the non-rigid transformation matrix V to determine the matching features corresponding to highly similar pathological images. Since the non-rigid transformation matrix V satisfies the angle error requirement, the matching features output by the similarity determination unit satisfy the angle error requirement and are derived from highly similar pathological images.

[0081] The fusion unit is used to fuse multiple matching features that meet the requirements of angle error and similarity to obtain the matching result of the pathological image.

[0082] The mouse unit is used to determine the coordinates of the center point of the area of ​​interest selected by the user.

[0083] The non-rigid transformation matrix D determination unit calculates the non-rigid transformation matrix D based on the matching results provided by the fusion unit. The non-rigid transformation matrix D is mapped to its coordinates on different pathological images based on the center point coordinates provided by the user. Using the mapped coordinates as the center point of the region of interest, the region of interest is extracted and, together with the region of interest selected by the user, constitutes the common region of the pathological image.

[0084] In summary, compared with existing methods, the technical solutions disclosed in this invention offer the following advantages: by using the pathological image matching method and apparatus provided by this invention, the interference of modal differences on the matching results can be effectively avoided. Furthermore, by fusing multi-matching channel matching features, it can better meet the clinical need for precise localization of common regions in pathological images for cancer auxiliary diagnosis. In addition, the pathological image matching apparatus supports the reuse of common regions, which can improve the accuracy and efficiency of cancer diagnosis. In conclusion, this technology has broad application prospects in the medical field and is expected to bring revolutionary improvements to pathological image processing and diagnosis. The advantages of this invention include:

[0085] Constructing multiple matching channels improves the reliability of pathological image matching features;

[0086] Fusion of multi-matching channel features supports accurate localization of common regions in pathological images;

[0087] The invention of a pathological image matching device allows for the reuse of common areas in pathological images, improving the efficiency of clinical diagnosis. Attached Figure Description

[0088] Figure 1 : A schematic diagram of the method flow of an embodiment of the present invention.

[0089] Figure 2 : A schematic diagram of the original pathological image in the image registration task of this embodiment of the invention.

[0090] Figure 3 : A schematic diagram of the matching channel construction in the image registration task of this invention embodiment.

[0091] Figure 4 : A schematic diagram of matching feature acquisition in the image registration task of this invention embodiment.

[0092] Figure 5 : A schematic diagram of the fusion matching result in the image registration task of this embodiment of the invention.

[0093] Figure 6 : Schematic diagram of a pathological image matching device in the image registration task of this invention embodiment. Detailed Implementation

[0094] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0095] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0096] See Figures 1 to 6 The present invention is illustrated in the field of image registration, and is described in detail below:

[0097] like Figure 1 As shown, the main steps of the embodiments provided by the present invention are as follows:

[0098] Step 1, normalize image modalities to construct matching channels:

[0099] (1-1) with Figure 2 Taking the three original pathological images shown as H&E, IHC (CD4), and IHC (CD20) as examples, the corresponding original pathological image G is represented. A Original pathological image G B Original pathological image G C ,like Figure 3 As shown, the CycleGAN deep learning-based image modality transformation network is used to normalize image modalities.

[0100] For pathological images G A G B G C Group into pairs. Using image G... A and image G B Taking the group as an example, first divide the image G into blocks. A and image G B A training set of 500,000 256×256 pixel blocks was obtained to train the image modality transfer network CycleGAN. Then, the trained CycleGAN network was used to transform the image G... A and image G B The pixel blocks are then stitched together to generate image L. A and image L B They are respectively related to image G B and image G A Modal consistency. Similarly, in image G A and image G C and image G B and image G C The group they belong to can also obtain modally consistent images.

[0101] (1-2) Each matching channel contains a pair of unmatched images GL, one being the original pathological image without modality transformation, and the other being generated by the image modality transformation network CycleGAN. In this embodiment, image G represents the original pathological image without modality transformation, and image L represents the image generated by CycleGAN.

[0102] (1-3) The image GL in each matching channel has a single modality, that is, the image G or L generated by the image modality conversion network has the same image modality as the original pathological image L or G.

[0103] (1-4) Perform image GL matching in the same matching channel; image GL matching is not required in different matching channels.

[0104] (1-5) The same or different matching algorithms are used in different matching channels. SIFT algorithm is used to match pathological images in matching channels 1 to 3, while SURF algorithm is used in matching channels 4 to 6.

[0105] Step 2, as follows Figure 4 As shown, image matching is performed separately in each matching channel to obtain multi-matching channel matching features:

[0106] (2-1) Use a matching algorithm to match the image GL in multiple matching channels, and each matching channel generates a set of matching features.

[0107] First, feature points are extracted from images G and L in each matching channel. These feature points are also called key points or interest points. In this embodiment, SIFT and SURF feature points are used for matching channels 1 to 3 and 4 to 6, respectively.

[0108] Then, SIFT and SURF descriptors are used to construct feature vectors for images G and L in matching channels 1 to 3 and 4 to 6, respectively, to describe feature points.

[0109] Finally, within each matching channel, the Euclidean distance of the feature vectors is calculated to measure the similarity between different feature points. The minimum Euclidean distance indicates high similarity between different feature points and is used to determine the matching features.

[0110] (2-2) The matching features under each matching channel are composed of matching feature point pairs (X p ,Y q M p N q ) indicates that (X) p ,Y q ) and (M p N q ) are the pixel coordinates of image G and image L, respectively. p,q are the pixel dimensions of image GL.

[0111] Step 3, as follows Figure 5 As shown, based on the consistency of the mapping relationship of matching features, the matching results are determined by fusing matching features from multiple matching channels:

[0112] (3-1) The mapping relationship of the image GL is represented by rigid transformation matrices and non-rigid transformation matrices, which are obtained by prediction based on the matching features of multiple matching channels. Specifically, they can be obtained by calling the rigid transformation matrix and non-rigid transformation matrix functions of the python-cv2 library. The matching features of each matching channel can generate a rigid transformation matrix and a non-rigid transformation matrix.

[0113] (3-2) The consistency of the mapping relationship of the image GL is determined by the angular error of the relative angle of the image GL between different matching channels and the similarity of the common region of the image GL in the same matching channel.

[0114] The rigid transformation matrix is ​​used to calculate the relative angle of the image GL. The relative angle represents the angular error of the image GL.

[0115] The calculation method for the angle error includes the following sub-steps:

[0116] The rigid transformation matrix predicted based on multi-match channel matching features is as follows:

[0117]

[0118] Among them, R n Let represent the rigid transformation matrix. Let n represent the number of matching channels. The relative angle θ of the image GL in each matching channel is given by the following formula:

[0119]

[0120]

[0121]

[0122] θ=[θ x ;θ y ;θ z ]

[0123] Where, θ x ,θ y ,θ z These represent the rotation angles of the pathological image around the X, Y, and Z axes, respectively. The formula for calculating the angle error is as follows:

[0124]

[0125] Among them, E iThis represents the angular error of the relative angle of image GL in the i-th matching channel. i∈[1,2,3,…,n] represents the order of the matching channels. For ease of description, it is defined as follows:

[0126]

[0127] in This represents the angular error in the X, Y, Z axes between the relative angle of image GL in the i-th matching channel and the relative angle of image GL in the n-th matching channel.

[0128] The angle error is less than the threshold T1. The smaller T1 is, the smaller the angle error, which indicates that the consistency of the matching features of the image GL between different matching channels is better.

[0129] The non-rigid transformation matrix is ​​used to locate common regions of images GL. These common regions are then used to calculate the similarity of images GL within the same matching channel.

[0130] The similarity is calculated using the following sub-steps:

[0131] Calculate the data H required to plot the angle error histogram.

[0132] H i =sum(E i <T1)

[0133] Among them, E i T1 represents the angular error of the relative angle of image GL in the i-th matching channel. T1 is the angular error threshold. i∈[1,2,3,…,n] represents the order of the matching channels.

[0134] Determine the order j of the matching channels corresponding to the most frequent histogram.

[0135] H i＝j =max(H)

[0136] Where H represents the data from the angle error histogram. j represents the most frequent matching channel. i∈[1,2,3,…,n] indicates the order of the matching channels.

[0137] The matching channel that has the same mapping relationship with the j-th matching channel is determined based on the matching characteristics of the j-th matching channel.

[0138] U = E j <T1

[0139] in, Therefore, U represents the index of all matching channels with consistent mapping relationships. E j T1 represents the angular error of the relative angle of image GL in the most frequently matched channel j. T1 is the angular error threshold. This represents the angular error in the X, Y, Z axes between the relative angle of image GL in the most frequent matching channel j and the relative angle of image GL in the nth matching channel.

[0140] Transform image L using a non-rigid transformation matrix to generate image W that shares a common region with image G.

[0141] W i =V i L i ,if U i =1

[0142] Where W is the image W. V is the non-rigid transformation matrix. L is the image L. i∈[1,2,3,…,n] represents the order of the matching channels.

[0143] Calculate the similarity level between image G and image W.

[0144]

[0145] Where S represents the similarity level of images GL in matching channels with consistent mapping relationships. f is the mean squared error, a metric for measuring image similarity. G is the image G.

[0146] The similarity is determined by image similarity metrics, including but not limited to the use of structural similarity index or mean squared error. The similarity of common regions is less than a threshold T2, and the smaller T2 is, the higher the GL similarity of images in the same matching channel.

[0147] (3-3) When there is only one pathological image matching channel, the matching feature is directly used as the matching result.

[0148] (3-4) When there are multiple pathological image matching channels, in order to achieve the purpose of fusing matching features from multiple matching channels, the angular error between the relative angle of image GL in each matching channel and the relative angle of image GL in all matching channels is calculated, and the histogram distribution of angle errors less than threshold T1 is statistically analyzed. The similarity level of image GL in the matching channel corresponding to the most frequent histogram is calculated, and the matching features with similarity levels less than threshold T2 are fused. The result is the matching result.

[0149] The process of fusing multi-matching channel matching features to obtain matching results also includes the following sub-steps:

[0150] K = S <T2

[0151]

[0152] Where K represents the matching channel index where the similarity level of image GL is less than T2. ​​S represents the similarity level of image GL in matching channels with consistent mapping relationships. T2 is the similarity threshold of the common region. F is the matching result fused with matching features from multiple matching channels.

[0153] Step 4: Based on the matching results, assist clinicians in accurately locating local common regions in pathological images:

[0154] (4-1) The user uses the mouse to select a region of interest on the pathological image to determine the coordinates of the center point of the region of interest.

[0155] (4-2) Based on the matching results obtained in step 3, calculate the non-rigid transformation matrix D. Specifically, it can be obtained by calling the non-rigid transformation matrix function of the python-cv2 library.

[0156] (4-3) Calculate the mapped coordinates of the center point of the region of interest in other corresponding pathological images based on the non-rigid transformation matrix D. Specifically, the calculation process involves multiplying the center point coordinates (x, y) of the region of interest with the non-rigid transformation D to obtain new coordinates (x', y'), which are the mapped coordinates to other corresponding pathological images.

[0157] (4-4) Using the mapped coordinates as the center point of the region of interest, the region of interest is extracted and together with the region of interest selected by the user to form the common region of the pathological image.

[0158] As mentioned above, [the text has already been referenced]. Figures 1 to 5 The specific process of the multi-channel fusion matching method for pathological images is described in detail. In the following text, reference will be made to... Figure 6 Describe the specific configuration of the pathological image matching device.

[0159] The matching channel unit is used to transform the original pathological image modality and construct a pathological image matching channel. The transformed image is used to construct a single-modality matching channel with the original pathological image (which is modally consistent).

[0160] The storage unit is used to store the matching features of the matching channels of pathological images.

[0161] The rigid transformation matrix determination unit is used to calculate the rigid transformation matrix obtained from the matching features within each matching channel.

[0162] The angle error determination unit is used to determine the relative angle error between different pathological images. It calculates the relative angle of the pathological images in each matching channel, compares the errors between the relative angles in different matching channels, and uses a threshold to determine the matching features that meet the angle error requirements.

[0163] The non-rigid transformation matrix V determination unit is used to calculate the non-rigid transformation matrix V obtained by the matching features within the matching channel that satisfy the angle error requirement. The non-rigid transformation matrix V is calculated using matching features that satisfy the angle error (defined by the angle error determination unit) (stored in the storage unit).

[0164] The similarity determination unit transforms the pathological image using a non-rigid transformation matrix V to determine the matching features corresponding to highly similar pathological images. Since the non-rigid transformation matrix V satisfies the angle error requirement, the matching features output by the similarity determination unit also satisfy the angle error requirement and are derived from highly similar pathological images.

[0165] The fusion unit is used to fuse multiple matching features that meet the requirements of angle error and similarity to obtain the matching results of pathological images.

[0166] The mouse unit is used to determine the coordinates of the center point of the area of ​​interest selected by the user.

[0167] The non-rigid transformation matrix D determination unit calculates the non-rigid transformation matrix D based on the matching results provided by the fusion unit. The non-rigid transformation matrix D is then mapped to its coordinates on different pathological images based on the center point coordinates provided by the user. Using the mapped coordinates as the center point of the region of interest, the region of interest is extracted and, together with the user-selected region of interest, constitutes the common region of the pathological image.

[0168] As mentioned above, [the text has already been referenced]. Figure 6 The specific configuration of the pathological image matching device is described in detail. The matching device provided by this invention has the ability to store multiple matching channels and effectively match features, thereby supporting efficient localization of common regions in pathological images.

[0169] As shown in Table 1, compared to traditional manual matching methods, this device offers greater objectivity and superior performance for clinical cases requiring frequent localization of common areas in pathological images. In terms of time cost, the matching device takes only 37±6 seconds, while manual matching (10 people) takes 647±68 seconds, significantly improving matching efficiency. Furthermore, in terms of matching accuracy, the device achieves a high accuracy of 84±58 micrometers, significantly better than the 112±31 micrometers achieved by manual matching (10 people). Overall, the matching device provided by this invention not only has a significant advantage in efficiency but also exhibits comparable or even better performance in matching accuracy, providing a satisfactory matching solution for diagnosing cancer using pathological images.

[0170] Table 1: Performance Statistics of Pathological Image Matching Devices in Image Registration Tasks

[0171]

[0172] This embodiment discloses an automatic matching method and system for multi-channel fusion of pathological images, which can accurately locate common regions in pathological images. Simultaneously, the pathological image matching device, by providing the function of reusing common regions in pathological images, can assist pathologists in cancer diagnosis.

[0173] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0174] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for automatic matching of multi-channel fusion in pathological images, characterized in that, Includes the following steps: Step 1: Acquire pathological images in multiple modalities. Using the pathological images in each modality as a baseline, normalize the remaining pathological images to the corresponding modality, resulting in multiple images for each modality, including: Step 1.1: Normalize image modalities to construct matching channels; Modality conversion algorithms are used to convert images of different modalities to a consistent modality in order to construct matching channels; Step 1.2: Each matching channel contains a pair of unmatched image GLs, one being the original pathological image without modality conversion, and the other being generated by the image modality conversion network; Step 1.3: The image GL in each matching channel has a single modality, that is, the image G or L generated by the image modality conversion network has the same image modality as the original pathological image L or G; Step 1.4: Perform image GL matching within the same matching channel; image GL matching is not required in different matching channels. Step 1.5: Perform feature matching on the image GL in each matching channel to obtain the matching feature point pairs corresponding to each matching channel; Step 2: Perform image matching on the images for each modality sequentially to obtain the matching features for each modality, including: Step 2.1: Use a matching algorithm to match the image GL across multiple matching channels, with each matching channel generating a set of matching features; Step 2.2: The matching features under each matching channel are composed of matching feature point pairs. It means that, among them and These are the pixel coordinates of image G and image L, respectively; The pixel size of the image in GL format; Step 3: Combine the matching features under each modality, and fuse the matching features under multiple modalities to obtain the fused matching features; Step 4: Based on the fused matching features, assist in the precise localization of common areas in pathological images.

2. The automatic matching method for multi-channel fusion of pathological images according to claim 1, characterized in that: Step 3 includes: Step 3.1: The mapping relationship of image GL is represented by rigid transformation matrix and non-rigid transformation matrix, wherein the rigid transformation matrix and non-rigid transformation matrix are obtained by prediction based on multi-matching channel matching features; It can be obtained by calling the rigid transformation matrix and non-rigid transformation matrix functions of the python-cv2 library; The matching features of each matching channel can generate a rigid transformation matrix and a non-rigid transformation matrix; Step 3.2: The consistency of the mapping relationship of the image GL is determined by the angular error of the relative angle of the image GL between different matching channels and the similarity of the common region of the image GL in the same matching channel; Step 3.3: When there is only one matching channel in the pathological image, the matching feature is directly used as the matching result; Step 3.4: When there are multiple pathological image matching channels, in order to achieve the purpose of fusing the matching features of multiple matching channels, calculate the angular error between the relative angle of image GL in each matching channel and the relative angle of image GL in all matching channels, and count the angle errors less than the threshold. The histogram distribution; calculate the similarity level of the image GL within the matching channel corresponding to the most frequent histogram; for similarity levels less than a threshold... The matching features are fused together, and the result is the matching result.

3. The automatic matching method for multi-channel fusion of pathological images according to claim 2, characterized in that: The consistency of the mapping relationship described in step 3 is characterized by including: The rigid transformation matrix is ​​used to calculate the relative angle of the image GL; the relative angle represents the angular error of the image GL. The angle error is less than the threshold. , The smaller the angle error, the better the consistency of the image GL matching features between different matching channels; The non-rigid transformation matrix represents the mapping relationship of the image GL, and therefore can be used to locate the common region of the image GL; specifically, the localization process involves locating the common region of interest by mapping the coordinates of the center point of the region of interest. Multiplying by the non-rigid transformation D yields the mapped coordinates. That is, the mapped coordinates to other corresponding pathological images, and then and Common regions can be located; common regions are used to calculate the similarity of images in the same matching channel (GL). The similarity is determined by image similarity metrics, including but not limited to using structural similarity index or mean squared error; the similarity of common regions is less than a threshold. , The smaller the value, the higher the GL similarity of the images in the same matching channel.

4. The automatic matching method for multi-channel fusion of pathological images according to claim 3, characterized in that: The angle error required for fusing multi-matching channel matching features in step 3 is as follows: The rigid transformation matrix predicted based on multi-match channel matching features is as follows: in, This represents the rigid transformation matrix; n represents the number of matching channels; , , This indicates the direction information of the image rotation along the x-axis; , , Indicates the y-axis direction information of the image rotation; , , This indicates the z-axis direction information of the image rotation; it also indicates the relative angle of the image GL in each matching channel of the rotation matrix. The formula is as follows: in, These represent the rotation angles of the pathological image around the X, Y, and Z axes, respectively; the formula for calculating the angle error is as follows: in, Indicates the first The angular error of the relative angle of the image GL in each matching channel; Indicates the order of the matched channels, defined for ease of description: in Indicates the first The angular error in the X, Y, Z axes between the relative angle of image GL in the nth matching channel and the relative angle of image GL in the nth matching channel.

5. The automatic matching method for multi-channel fusion of pathological images according to claim 4, characterized in that: Step 3, which involves fusing the similarity required for multi-channel matching features, is as follows: Calculate the data for plotting the angle error histogram. ; in, Indicates the first The angular error of the relative angle of the image GL in each matching channel; This is the angle error threshold; Indicates the order of the matched channels; Determine the order of matching channels corresponding to the most frequent histogram. ; in, This is the data from the angle error histogram; For the most frequent matching channel; Indicates the order of the matched channels; According to the The matching features of each matching channel determine the matching channels that are consistent with their mapping relationship. in, ,therefore Indicates the index of all matching channels with consistent mapping relationships; Indicates the most frequent matching channel The angular error of the relative angle of the GL image in the middle; This is the angle error threshold; Indicates the most frequent matching channel The angular error between the relative angle of the middle image GL and the relative angle of the nth matching channel image GL in the X, Y, Z axis directions; Image L is transformed using a non-rigid transformation matrix to generate an image W that shares a common region with image G; in, It is image W; It is a non-rigid transformation matrix; It is image L; Indicates the order of the matched channels; Calculate the similarity level between image G and image W; in, This indicates the similarity level of images in GL within matching channels with consistent mapping relationships; It is an indicator for measuring image similarity, including but not limited to using structural similarity index or mean square error; It is image G.

6. The automatic matching method for multi-channel fusion of pathological images according to claim 5, characterized in that: The fusion of multi-matching channel matching features described in step 3 further includes the following sub-steps: The matching results are obtained by fusing matching features from multiple matching channels; in, This indicates that the similarity level of the images in GL is less than The matching channel index; This indicates the similarity level of images in GL within matching channels with consistent mapping relationships; The similarity threshold for the common regions; It is a matching result that integrates matching features from multiple matching channels.

7. The automatic matching method for multi-channel fusion of pathological images according to claim 6, characterized in that: Step 4 includes: Step 4.1: The user uses the mouse to select a region of interest on the pathological image to determine the coordinates of the center point of the region of interest; Step 4.2: Calculate the non-rigid transformation matrix based on the matching results obtained in Step 3. D Specifically, it can be obtained by calling the non-rigid transformation matrix function of the python-cv2 library; Step 4.3: Based on the non-rigid transformation matrix D The mapping coordinates of the center point coordinates of the region of interest in other corresponding pathological images are calculated; specifically, the calculation process involves mapping the center point coordinates of the region of interest to the corresponding pathological images. Multiplying by the non-rigid transformation D yields new coordinates. That is, the mapped coordinates to other corresponding pathological images; Step 4.4: Using the mapped coordinates as the center point of the region of interest, extract the region of interest and combine it with the region of interest selected by the user to form a common area of ​​the pathological image.

8. A method system for automatic matching of multi-channel fusion of pathological images, characterized in that, include: Matching channel unit, storage unit, rigid transformation matrix determination unit, angle error determination unit, non-rigid transformation matrix V Determining unit, similarity determining unit, fusion unit, mouse unit, non-rigid transformation matrix D Define the unit; The matching channel unit is used to convert the original pathological image modality and construct pathological image matching channels, including the following operations: using a modality conversion algorithm to convert images of different modalities to a consistent modality to construct matching channels; each matching channel contains a pair of unmatched images GL, one being the original pathological image without modality conversion, and the other being generated by an image modality conversion network; each image GL in a matching channel has a single modality, that is, the image G or L generated by the image modality conversion network has the same image modality as the original pathological image L or G; image GL matching is performed in the same matching channel, and image GL in different matching channels does not need to be matched; feature matching is performed on the image GL in each matching channel to obtain matching feature point pairs corresponding to each matching channel; matching algorithms are used to match image GL in multiple matching channels, and each matching channel generates a set of matching features; the matching features in each matching channel are composed of matching feature point pairs. It means that, among them and These are the pixel coordinates of image G and image L, respectively; The pixel size of the image in GL format; The converted image and the original pathological image (modal consistency) are used to form a single-modal matching channel; The storage unit is used to store the matching features of the pathological image matching channel; The rigid transformation matrix determination unit is used to calculate the rigid transformation matrix obtained from the matching features in each matching channel; The angle error determination unit is used to determine the relative angle error between different pathological images; calculate the relative angle of the pathological image in each matching channel, compare the error between the relative angles in different matching channels, and use a threshold to determine the matching features that meet the angle error requirements; The non-rigid transformation matrix V The determining unit is used to calculate the non-rigid transformation matrix obtained by calculating the matching features within the matching channel that meet the angle error requirements. V Non-rigid transformation matrix V The matching features that satisfy the angle error are calculated and defined by the angle error determination unit and stored in the storage unit; The similarity determination unit determines the similarity based on the non-rigid transformation matrix. V Transform pathological images to determine matching features corresponding to highly similar pathological images; due to the non-rigid transformation matrix... V The angle error requirement is met, therefore the matching features output by the similarity determination unit meet the angle error requirement and are matched from highly similar pathological images; The fusion unit is used to fuse multiple matching features that meet the requirements of angle error and similarity to obtain the matching result of the pathological image; The mouse unit is used to determine the coordinates of the center point of the area of ​​interest selected by the user. The non-rigid transformation matrix D The determining unit calculates the non-rigid transformation matrix based on the matching results provided by the fusion unit. D Non-rigid transformation matrix D The coordinates of the center point provided by the user are mapped to the coordinates of the center point on different pathological images; the mapped coordinates are used as the center point of the region of interest, the region of interest is extracted, and together with the region of interest selected by the user, they form the common region of the pathological image.

Images