A method and system for multi-resolution digital pathology image registration

Abstract

The present invention provides a method and system for multi-resolution digital pathology image registration, wherein the method comprises: step 1: obtaining a source image and a target image, wherein the source image and the target image are multi-resolution digital pathology images obtained by scanning the same pathology slice once by two different scanners; step 2: at a low-resolution level, registering the source image and the target image to obtain a first transformation matrix; step 3: at a high-resolution level, based on the first transformation matrix, determining a second region in the source image corresponding to each first region of the target image, registering the first region and the second region to obtain a second transformation matrix; step 4: obtaining a registration result based on the first transformation matrix and the second transformation matrix. The multi-resolution digital pathology image registration method of the present invention reduces the amount of calculation during registration, reduces time consumption, and improves registration accuracy by combining low-resolution global registration with high-resolution local registration.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system suitable for multi-resolution digital pathology image registration. Background Art

[0002] With the development of computer technology in the field of medical image processing, digital images are playing an increasingly important role in pathological analysis and diagnosis.

[0003] Image registration is a very important step in the processing of digital pathology sections. For example, in some cases, it is necessary to compare and analyze sections with different staining such as H&E and IHC. Without registration, it is difficult to locate common areas in multiple images. In the application of artificial intelligence, data annotation is a time-consuming and laborious task. The application of image registration technology can migrate annotations of images of the same section from different scanners, thereby reducing a lot of annotation work.

[0004] Digital pathological images have the characteristics of multi-level, high resolution and large image size. There are two main types of traditional registration methods: the first is to implement the registration algorithm on the entire image. This method requires processing every pixel, but this method needs to process every pixel in the image. For high-resolution pathological images, the calculation is large, time-consuming, and prone to registration failure; the second is to register thumbnails through downsampling methods, and then apply the mapping relationship obtained by registration to the high-resolution image level. However, this method registers low-resolution thumbnails through downsampling, which greatly improves the calculation speed, but when migrating to the high-resolution image level, the registration error is amplified and the accuracy is reduced. Summary of the invention

[0005] The present invention provides a method and system for multi-resolution digital pathology image registration. By combining low-resolution global registration with high-resolution local registration, the amount of calculation during registration is reduced, the time consumption is reduced, and the registration accuracy is improved. The method and system are suitable for high-resolution digital pathology image registration problems, and can solve the registration problems between digital pathology images with different staining such as H&E and IHC. The system can also be used for image annotation migration.

[0006] The present invention provides a multi-resolution digital pathology image registration method, comprising:

[0007] Step 1: Acquire a source image and a target image, wherein the source image and the target image are multi-resolution digital pathology images obtained by scanning the same pathology section once by two different scanners;

[0008] Step 2: At the low-resolution level, align the source image and the target image to obtain the first transformation matrix;

[0009] Step 3: at a high-resolution level, based on the first transformation matrix, determining a second region in the source image corresponding to each first region of the target image, registering the first region and the second region, and obtaining a second transformation matrix;

[0010] Step 4: Based on the first transformation matrix and the second transformation matrix, obtain the registration result.

[0011] Preferably, in step 2, registering the source image and the target image includes:

[0012] Performing grayscale conversion on the source image to obtain a first grayscale image;

[0013] Performing feature extraction on the first grayscale image to obtain first feature vectors of a plurality of first key points;

[0014] Performing grayscale conversion on the target image to obtain a second grayscale image;

[0015] Performing feature extraction on the second grayscale image to obtain second feature vectors of multiple second key points;

[0016] Matching the first eigenvector with the second eigenvector;

[0017] A first transformation matrix is ​​determined based on the first coordinates of the first key point and the second coordinates of the second key point corresponding to the first eigenvector and the second eigenvector that match each other.

[0018] Preferably, in step 3, registering the first region and the second region includes:

[0019] Performing grayscale conversion on the first region to obtain a third grayscale image;

[0020] Performing feature extraction on the third grayscale image to obtain third feature vectors of a plurality of third key points;

[0021] Performing grayscale conversion on the second region to obtain a fourth grayscale image;

[0022] Performing feature extraction on the fourth grayscale image to obtain fourth feature vectors of a plurality of fourth key points;

[0023] Matching the third eigenvector and the fourth eigenvector;

[0024] A second transformation matrix is ​​determined based on the third coordinates of the third key point and the fourth coordinates of the fourth key point corresponding to the third eigenvector and the fourth eigenvector that match the matching.

[0025] Preferably, step 4: obtaining a registration result based on the first transformation matrix and the second transformation matrix includes:

[0026] Output display source image;

[0027] Acquire a sight line change of a first user within a recently preset first time period when viewing a source image;

[0028] Based on the change of sight line, determine the moving trajectory of the sight line on the source image;

[0029] Extracting features from the moving trajectory based on a preset first feature extraction template to obtain a plurality of first feature values;

[0030] Based on the first eigenvalue, construct a trajectory description vector;

[0031] Get the preset trigger vector library;

[0032] Matching the trajectory description vector with the trigger vector in the trigger vector library;

[0033] If the match is satisfied, determining at least one viewing area viewed by the user on the source image based on the movement trajectory;

[0034] determining a registration region on the target image corresponding to the viewing region based on the first transformation matrix and the second transformation matrix;

[0035] Based on the movement trajectory, determining a mapping area on the source image suitable for the registration area;

[0036] Map the registration area into the mapping area, obtain the registration result, and complete the acquisition.

[0037] Preferably, determining at least one viewing area viewed by the user on the source image based on the movement trajectory includes:

[0038] Acquire a plurality of first area blocks that the moving trajectory passes through in the source image;

[0039] Traverse the first area block in sequence;

[0040] During each traversal, a local trajectory of the moving trajectory falling within the traversed first area block is extracted, and at the same time, a second area block adjacent to the traversed first area block in the source image except for other first area blocks is selected;

[0041] Based on a preset second feature extraction template, feature extraction is performed on the traversed first area block and the second area block and the relative position between the local trajectory and the first boundary line adjacent to the traversed first area block to obtain a plurality of second feature values;

[0042] Based on the second eigenvalue, construct an association description vector;

[0043] Get the preset correlation vector library;

[0044] Matching the association description vector with the association vector in the association vector library;

[0045] If the match is met, the corresponding second area block is used as the third area block, and at the same time, the third area block is marked as unselectable;

[0046] The first area block and the third area block are integrated to obtain a viewing area viewed by the user on the source image, and the determination is completed.

[0047] Preferably, determining a mapping area on the source image suitable for the registration area based on the movement trajectory includes:

[0048] Randomly mapping the registration area in the source image, and recording a plurality of first areas randomly mapped;

[0049] The first area which has no overlap with the viewing area is used as the second area;

[0050] Based on a preset third feature extraction template, feature extraction is performed on the relative position between the second area and the first area to obtain a plurality of third feature values;

[0051] Based on the third eigenvalue, construct the analysis description vector;

[0052] Acquire at least one preset analysis habit vector corresponding to the first user;

[0053] Calculate the similarity between the analysis description vector and the analysis habit vector;

[0054] If the similarity is greater than or equal to a preset similarity threshold, the corresponding second area is used as the third area;

[0055] Based on a preset fourth feature extraction template, feature extraction is performed on a preset range around a plurality of points on a second boundary line of a third region in the source image to obtain a plurality of fourth feature values;

[0056] Based on the fourth eigenvalue, construct a difference description vector;

[0057] Obtaining a preset library of suitable evaluation vectors;

[0058] Determine the suitability based on the difference description vector and the suitability evaluation vector library;

[0059] The third region corresponding to the maximum suitability is selected as the mapping region on the source image suitable for the registration region, and the determination is completed.

[0060] Preferably, the multi-resolution digital pathology image registration method further includes:

[0061] Obtaining analysis reference content corresponding to the viewing area and the registration area, and adding the analysis reference content to the registration result;

[0062] The obtaining of analysis reference content corresponding to the viewing area and the registration area includes:

[0063] Obtaining a preset analysis model corresponding to the slice type of the pathological slice;

[0064] The viewing area and the registration area are input into the analysis model, the analysis reference content is determined, and the acquisition is completed.

[0065] Preferably, the multi-resolution digital pathology image registration method further includes:

[0066] When the first user requests at least one second user to jointly analyze the registration result, an online analysis room is constructed, and the first user and the second user are connected to the online analysis room;

[0067] Public registration results in the online analysis room;

[0068] Acquire multiple conversation records generated when the first user and the second user perform online analysis;

[0069] Dynamically adjust the registration results based on the conversation records;

[0070] Among them, based on the conversation record, the registration result is dynamically adjusted, including:

[0071] Construct a timeline;

[0072] Based on the time point when the conversation record was generated, the conversation record is correspondingly set on the time axis;

[0073] Perform semantic extraction on the conversation record to obtain the first semantics;

[0074] Obtaining a preset adjustment instruction semantic library;

[0075] matching the first semantics with the second semantics in the adjustment instruction semantic library;

[0076] If the match is met, based on the semantic understanding technology, based on the first semantics of a preset number of conversation records after the conversation record corresponding to the first semantics that matches the match on the time axis, determine whether the preset adjustment operation corresponding to the second semantics that matches the match is effective;

[0077] If so, in response to the adjustment operation, the registration result is adjusted accordingly.

[0078] The present invention provides a multi-resolution digital pathology image registration system, comprising:

[0079] A first acquisition module is used to acquire a source image and a target image, wherein the source image and the target image are multi-resolution digital pathology images obtained by scanning the same pathology slice once by two different scanners;

[0080] A first registration module, used for registering the source image and the target image at a low resolution level to obtain a first transformation matrix;

[0081] A second registration module is used to determine, at a high-resolution level, a second region in the source image corresponding to each first region of the target image based on the first transformation matrix, and to register the first region and the second region to obtain a second transformation matrix;

[0082] The second acquisition module is used to acquire the registration result based on the first transformation matrix and the second transformation matrix.

[0083] Preferably, the first registration module registers the source image and the target image, including:

[0084] Performing grayscale conversion on the source image to obtain a first grayscale image;

[0085] Performing feature extraction on the first grayscale image to obtain first feature vectors of a plurality of first key points;

[0086] Performing grayscale conversion on the target image to obtain a second grayscale image;

[0087] Performing feature extraction on the second grayscale image to obtain second feature vectors of multiple second key points;

[0088] Matching the first eigenvector with the second eigenvector;

[0089] A first transformation matrix is ​​determined based on the first coordinates of the first key point and the second coordinates of the second key point corresponding to the first eigenvector and the second eigenvector that match each other.

[0090] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0091] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0093] Figure 1 A flowchart of a multi-resolution digital pathology image registration method according to an embodiment of the present invention;

[0094] Figure 2This is a process example diagram of a multi-resolution digital pathology image registration method in an embodiment of the present invention;

[0095] Figure 3 is another process example diagram applicable to a multi-resolution digital pathology image registration method in an embodiment of the present invention;

[0096] Figure 4 FIG. 4 is another process example diagram of a multi-resolution digital pathology image registration method according to an embodiment of the present invention;

[0097] Figure 5 Schematic diagram of a multi-resolution digital pathology image registration system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0098] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0099] The present invention provides a multi-resolution digital pathology image registration method, such as Figure 1 As shown, including:

[0100] Step 1: Acquire a source image and a target image, wherein the source image and the target image are multi-resolution digital pathology images obtained by scanning the same pathology section once by two different scanners;

[0101] Step 2: At the low-resolution level, align the source image and the target image to obtain the first transformation matrix;

[0102] Step 3: at a high-resolution level, based on the first transformation matrix, determining a second region in the source image corresponding to each first region of the target image, registering the first region and the second region, and obtaining a second transformation matrix;

[0103] Step 4: Based on the first transformation matrix and the second transformation matrix, obtain the registration result.

[0104] The working principle and beneficial effects of the above technical solution are:

[0105] The scanner can be a fully automatic microscope and an optical magnification system. Two different scanners scan the same pathological slice to obtain two multi-resolution pyramid images, namely multi-resolution digital pathological images, which are used as the source image and the target image respectively. Figure 2As shown in Figure 1, at the low-resolution level, the source image (left image) and the target image (right image) are registered to obtain the transformation matrix H1, i.e., the first transformation matrix. The first transformation matrix represents the mapping relationship between the point coordinates on the low-resolution source image and the point coordinates on the target image. As the resolution increases, the registration error becomes more obvious, as shown in Figure 1. Figure 3 As shown in (a) and (b) in FIG. 1 , at a high-resolution level, a region, i.e., the first region, is read from the target image, and another region, i.e., the second region, corresponding to the region is read from the source image according to the transformation matrix H1. The registration is performed again to obtain the transformation matrix H2, i.e., the second transformation matrix. The second transformation matrix represents the mapping relationship between the point coordinates on the first region and the point coordinates on the second region at a high-resolution level. Figure 4 As shown, according to the first transformation matrix and the second transformation matrix, any area of ​​the source image can be mapped to the target image to obtain the registration result. By combining low-resolution global registration with high-resolution local registration, the amount of calculation during registration is reduced, the time consumption is reduced, and the registration accuracy is improved. It is suitable for high-resolution digital pathology image registration problems, solves the registration problem between digital pathology images with different staining such as H&E and IHC, and can also be used for image annotation migration.

[0106] The present invention provides a multi-resolution digital pathology image registration method, wherein in step 2, the source image and the target image are registered, including:

[0107] Performing grayscale conversion on the source image to obtain a first grayscale image;

[0108] Performing feature extraction on the first grayscale image to obtain first feature vectors of a plurality of first key points;

[0109] Performing grayscale conversion on the target image to obtain a second grayscale image;

[0110] Performing feature extraction on the second grayscale image to obtain second feature vectors of multiple second key points;

[0111] Matching the first eigenvector with the second eigenvector;

[0112] A first transformation matrix is ​​determined based on the first coordinates of the first key point and the second coordinates of the second key point corresponding to the first eigenvector and the second eigenvector that match each other.

[0113] The working principle and beneficial effects of the above technical solution are:

[0114] The source image and the target image are converted into grayscale images respectively. Feature extraction is performed on the two grayscale images to extract two groups of key points and feature vectors of the key points, where the feature vectors are image feature vectors of the key points. Feature matching is performed on the feature vectors of the two groups of key points, and the first transformation matrix is ​​determined according to the matched key point coordinates. Grayscale conversion, feature extraction, and feature matching all belong to the scope of the prior art and will not be described in detail.

[0115] The present invention provides a multi-resolution digital pathology image registration method, wherein in step 3, the first region and the second region are registered, including:

[0116] Performing grayscale conversion on the first region to obtain a third grayscale image;

[0117] Performing feature extraction on the third grayscale image to obtain third feature vectors of a plurality of third key points;

[0118] Performing grayscale conversion on the second region to obtain a fourth grayscale image;

[0119] Performing feature extraction on the fourth grayscale image to obtain fourth feature vectors of a plurality of fourth key points;

[0120] Matching the third eigenvector and the fourth eigenvector;

[0121] A second transformation matrix is ​​determined based on the third coordinates of the third key point and the fourth coordinates of the fourth key point corresponding to the third eigenvector and the fourth eigenvector that match the matching.

[0122] The working principle and beneficial effects of the above technical solution are:

[0123] The principle of registering the first region with the second region is the same as the principle of registering the source image with the target image, and will not be elaborated herein.

[0124] The present invention provides a multi-resolution digital pathology image registration method, step 4: based on the first transformation matrix and the second transformation matrix, obtaining a registration result, including:

[0125] Output display source image;

[0126] Acquire a sight line change of a first user within a recently preset first time period when viewing a source image;

[0127] Based on the change of sight line, determine the moving trajectory of the sight line on the source image;

[0128] Extracting features from the moving trajectory based on a preset first feature extraction template to obtain a plurality of first feature values;

[0129] Based on the first eigenvalue, construct a trajectory description vector;

[0130] Get the preset trigger vector library;

[0131] Matching the trajectory description vector with the trigger vector in the trigger vector library;

[0132] If the match is satisfied, determining at least one viewing area viewed by the user on the source image based on the movement trajectory;

[0133] determining a registration region on the target image corresponding to the viewing region based on the first transformation matrix and the second transformation matrix;

[0134] Based on the movement trajectory, determining a mapping area on the source image suitable for the registration area;

[0135] Map the registration area into the mapping area, obtain the registration result, and complete the acquisition.

[0136] The working principle and beneficial effects of the above technical solution are:

[0137] Generally, when researchers want to obtain registration results for comparative analysis, they will circle the source image. After the circle is completed, another area in the target image corresponding to the circled area is determined according to the first transformation matrix and the second transformation matrix, and then the comparative analysis is conducted. However, the circle is based on the actual needs of the researchers, so the circle range is often irregular, and multiple circle selections are required for one analysis. The images after scanning the pathological sections are often complex, and the researchers have to circle them all, which is less convenient and reduces the efficiency of the analysis. In particular, when the number of circle selections is large, it may cause eye discomfort to the researchers. Therefore, it is urgent to solve this problem.

[0138] Output and display the source image, and the first user views the source image on the display screen. Obtain the sight line change of the first user in the most recent first time when viewing the source image. The first time can be 20 seconds. The sight line acquisition and sight line change belong to the scope of the prior art and will not be elaborated. Based on the sight line change, the moving trajectory of the sight line falling on the source image in the display screen can be determined. The moving trajectory is a trajectory composed of the points where the sight line falls on the display screen. Introduce a preset first feature extraction template, extract features from the moving trajectory, and obtain multiple first feature values. The first feature value can be the straight-line distance, overlap, and proportion of the trajectory to the same area block at the beginning and end of the trajectory. Based on the first feature value, construct a trajectory description vector. Constructing a vector based on data belongs to the scope of the prior art and will not be elaborated. Introduce a preset trigger vector library, which stores a large number of vectors composed of determining the moving trajectory based on the sight line change when the user is looking at a certain area and extracting the feature value (the process is the same as the construction process of the above-mentioned trajectory description vector). Match the trajectory description vector with the trigger vector therein. If the match is consistent, it means that the user is looking at a certain area, and start to identify the viewing area of ​​the gaze. It greatly reduces the recognition resources for gaze area recognition, reduces power consumption, and is more intelligent.

[0139] When performing gaze area recognition, it can be determined based on the movement trajectory. After determining the viewing area, the region mapping relationship between the source image and the target image is determined based on the first transformation matrix and the second transformation matrix, and the registration area is determined for mapping. There is no need for users to circle, which improves convenience and analysis and research efficiency, especially when the workload of analysis and research is large, it is more applicable and can avoid possible eye discomfort caused by researchers.

[0140] The present invention provides a multi-resolution digital pathology image registration method, which determines at least one viewing area viewed by a user on a source image based on a movement trajectory, comprising:

[0141] Acquire a plurality of first area blocks that the moving trajectory passes through in the source image;

[0142] Traverse the first area block in sequence;

[0143] During each traversal, a local trajectory of the moving trajectory falling within the traversed first area block is extracted, and at the same time, a second area block adjacent to the traversed first area block in the source image except for other first area blocks is selected;

[0144] Based on a preset second feature extraction template, feature extraction is performed on the traversed first area block and the second area block and the relative position between the local trajectory and the first boundary line adjacent to the traversed first area block to obtain a plurality of second feature values;

[0145] Based on the second eigenvalue, construct an association description vector;

[0146] Get the preset correlation vector library;

[0147] Matching the association description vector with the association vector in the association vector library;

[0148] If the match is met, the corresponding second area block is used as the third area block, and at the same time, the third area block is marked as unselectable;

[0149] The first area block and the third area block are integrated to obtain a viewing area viewed by the user on the source image, and the determination is completed.

[0150] The working principle and beneficial effects of the above technical solution are:

[0151] The source image is divided into n blocks of equal size. The first block through which the moving trajectory passes in the source image is obtained. Passing means that there is part of the moving trajectory in the first block. The first block is traversed in sequence. Each time the traversal is performed, the local track of the moving trajectory falling in the traversed first block is extracted. At the same time, the second block adjacent to the traversed first block is selected except for other first blocks. A preset second feature extraction template is introduced to extract features of the relative position between the traversed first block and the second block and the local track and the first boundary line, and obtain multiple second feature values. The second feature value can be a value representing the content features of the first block and the second block, and the distance between the point on the local track and the point on the first boundary line. Based on the second feature value, an association description vector is constructed. A preset association vector library is introduced, and the association vector library stores a large number of vectors indicating that there is an association between the contents of the two blocks and / or indicating that the user may view the two blocks together. The association description vector is matched with the association vector therein. If the match is met, the corresponding second block is used as the third block. Generally, due to the influence of peripheral vision, the line of sight acquisition may not be accurate. Therefore, based on the content relevance between the two area blocks and the characteristics of the relative position between the local track and the first boundary line (for example, the local track is very close to the first boundary line), the area that the user may still be looking at can be predicted. Improve the accuracy of viewing area acquisition. In addition, the purpose of marking the third area block as unselectable is to avoid repeated operations when traversing the first area block later.

[0152] The present invention provides a multi-resolution digital pathology image registration method, which determines a mapping area on a source image suitable for a registration area based on a movement trajectory, comprising:

[0153] Randomly mapping the registration area in the source image, and recording a plurality of first areas randomly mapped;

[0154] The first area which has no overlap with the viewing area is used as the second area;

[0155] Based on a preset third feature extraction template, feature extraction is performed on the relative position between the second area and the first area to obtain a plurality of third feature values;

[0156] Based on the third eigenvalue, construct the analysis description vector;

[0157] Acquire at least one preset analysis habit vector corresponding to the first user;

[0158] Calculate the similarity between the analysis description vector and the analysis habit vector;

[0159] If the similarity is greater than or equal to a preset similarity threshold, the corresponding second area is used as the third area;

[0160] Based on a preset fourth feature extraction template, feature extraction is performed on a preset range around a plurality of points on a second boundary line of a third region in the source image to obtain a plurality of fourth feature values;

[0161] Based on the fourth eigenvalue, construct a difference description vector;

[0162] Obtaining a preset library of suitable evaluation vectors;

[0163] Determine the suitability based on the difference description vector and the suitability evaluation vector library;

[0164] The third region corresponding to the maximum suitability is selected as the mapping region on the source image suitable for the registration region, and the determination is completed.

[0165] The working principle and beneficial effects of the above technical solution are:

[0166] The registration area is randomly mapped in the source image, and multiple first areas for random mapping are recorded. If there is no overlapping area between the first area and the viewing area, it means that when the registration area is mapped to the first area, it does not affect the researcher's comparative analysis based on the viewing area and the registration area of ​​the source image, as the second area. A preset third feature extraction template is introduced to extract features of the relative position between the second area and the first area, and multiple third eigenvalues ​​are obtained. The third eigenvalue can be a numerical value representing the relative direction and relative distance between the second area and the first area. Based on the third eigenvalue, an analysis description vector is constructed. At least one preset analysis habit vector corresponding to the first user is introduced. The analysis habit vector represents the positional relationship between the positions where the researchers are accustomed to placing the two images when performing comparative analysis. The similarity between the analysis description vector and the analysis habit vector is calculated. The greater the similarity, the more it satisfies the researcher's habit when mapping the registration area to the corresponding second area. In addition, a preset fourth feature extraction template is introduced to extract features from a preset range around multiple points on the second boundary line of the third area in the source image to obtain multiple fourth eigenvalues. The preset range can be a circular range with a radius of 2.5 cm, and the fourth eigenvalue can be a numerical value representing the image features within the trajectory within the surrounding range and the image features outside the trajectory. Based on the fourth eigenvalue, a difference description vector is constructed. A preset suitable evaluation vector library is introduced, and the suitable evaluation vector library stores the suitability corresponding to different difference description vectors. The greater the suitability, the more obvious the difference between the inside and outside of the trajectory within the surrounding range. The third area corresponding to the maximum suitability is selected as the mapping area on the source image suitable for the registration area to complete the determination. It not only ensures that the mapping area conforms to the habits of researchers as much as possible, improves user experience, and is more humane, but also ensures that the registration area is mapped to the mapping area, and the difference between the inside and outside of the trajectory is obvious, effectively avoiding the situation where the difference is not obvious and causes researchers to have difficulty in distinguishing. In addition, the formula for calculating the similarity between the analysis description vector and the analysis habit vector is: σ is the similarity, α and β are the analysis description vector and analysis habit vector respectively.

[0167] The present invention provides a multi-resolution digital pathology image registration method, which also includes:

[0168] Obtaining analysis reference content corresponding to the viewing area and the registration area, and adding the analysis reference content to the registration result;

[0169] The obtaining of analysis reference content corresponding to the viewing area and the registration area includes:

[0170] Obtaining a preset analysis model corresponding to the slice type of the pathological slice;

[0171] The viewing area and the registration area are input into the analysis model, the analysis reference content is determined, and the acquisition is completed.

[0172] The working principle and beneficial effects of the above technical solution are:

[0173] Obtaining analysis reference content corresponding to the viewing area and the registration area is not as good as the registration result. Researchers can make analysis references based on the analysis reference content to improve the efficiency of analysis research. When obtaining reference content, the preset analysis model corresponding to the slice type of the pathological section is introduced. The analysis model is a neural network model trained using a large amount of pathological section analysis experimental data and / or records of manual pathological section analysis. The viewing area and the registration area are input into the analysis model to determine the analysis reference content. The accuracy and efficiency of obtaining the analysis reference content are improved.

[0174] The present invention provides a multi-resolution digital pathology image registration method, which also includes:

[0175] When the first user requests at least one second user to jointly analyze the registration result, an online analysis room is constructed, and the first user and the second user are connected to the online analysis room;

[0176] Public registration results in the online analysis room;

[0177] Acquire multiple conversation records generated when the first user and the second user perform online analysis;

[0178] Dynamically adjust the registration results based on the conversation records;

[0179] Among them, based on the conversation record, the registration result is dynamically adjusted, including:

[0180] Construct a timeline;

[0181] Based on the time point when the conversation record was generated, the conversation record is correspondingly set on the time axis;

[0182] Perform semantic extraction on the conversation record to obtain the first semantics;

[0183] Obtaining a preset adjustment instruction semantic library;

[0184] matching the first semantics with the second semantics in the adjustment instruction semantic library;

[0185] If the match is met, based on the semantic understanding technology, based on the first semantics of a preset number of conversation records after the conversation record corresponding to the first semantics that matches the match on the time axis, determine whether the preset adjustment operation corresponding to the second semantics that matches the match is effective;

[0186] If so, in response to the adjustment operation, the registration result is adjusted accordingly.

[0187] The working principle and beneficial effects of the above technical solution are:

[0188] Generally, when the first user analyzes the registration result, he may invite the second user to analyze it together. When they analyze together, they may operate the registration result for easy viewing, such as zooming in and rotating, etc. However, each user often has permissions, and when making adjustments, due to network delays and other issues, adjustments may be confused. When only one user has permissions, other users need to request the authorized person to make adjustments, which is cumbersome. Therefore, a solution is urgently needed.

[0189] Construct an online analysis room, which belongs to the category of online conference technology. Obtain a conversation record and set it on the time axis. Extract the first semantics of the conversation record. Introduce a preset adjustment instruction semantic library, which stores a large number of semantics that are expected to be adjusted. Match the first semantics with the second semantics therein. If the match is consistent, it means that a user wants to make an adjustment operation. Based on the first semantics of a preset number of conversation records after the conversation record corresponding to the first semantics that matches on the time axis, determine whether the preset adjustment operation corresponding to the second semantics that matches is effective. For example, if the first semantics that matches is "enlarge 20 times", and then the first semantics of the preset number of conversation records contain "enlarge 15 times is enough", then the preset adjustment operation corresponding to the second semantics that matches is not effective. For another example, if the first semantics that matches is "reduce 10 times", and then the first semantics of the preset number of conversation records contain "reduce 10 times is about the same", then the preset adjustment operation corresponding to the second semantics that matches is effective. The preset number can be 5. The preset adjustment operation can be, for example, "enlarge 10 times", etc. If effective, respond to the adjustment operation and make corresponding adjustments to the registration result. There is no need for users to make adjustments by themselves, which avoids confusion or cumbersome adjustments when every user has permission or only one user has permission, improving convenience and user experience. In addition, the adjustment operation is confirmed to be effective, which improves the accuracy of semantic recognition-based operation instructions and is more intelligent.

[0190] The present invention provides a multi-resolution digital pathology image registration system, such as Figure 5 As shown, including:

[0191] A first acquisition module 1 is used to acquire a source image and a target image, wherein the source image and the target image are multi-resolution digital pathology images obtained by scanning the same pathology slice once by two different scanners;

[0192] A first registration module 2, used for registering the source image and the target image at a low resolution level to obtain a first transformation matrix;

[0193] A second registration module 3 is used to determine, at a high-resolution level, a second region in the source image corresponding to each first region of the target image based on the first transformation matrix, and to register the first region and the second region to obtain a second transformation matrix;

[0194] The second acquisition module 4 is used to acquire the registration result based on the first transformation matrix and the second transformation matrix.

[0195] The present invention provides a multi-resolution digital pathology image registration system, wherein a first registration module 2 registers a source image and a target image, and comprises:

[0196] Performing grayscale conversion on the source image to obtain a first grayscale image;

[0197] Performing feature extraction on the first grayscale image to obtain first feature vectors of a plurality of first key points;

[0198] Performing grayscale conversion on the target image to obtain a second grayscale image;

[0199] Performing feature extraction on the second grayscale image to obtain second feature vectors of multiple second key points;

[0200] Matching the first eigenvector with the second eigenvector;

[0201] A first transformation matrix is ​​determined based on the first coordinates of the first key point and the second coordinates of the second key point corresponding to the first eigenvector and the second eigenvector that match each other.

[0202] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-resolution digital pathology image registration method, characterized in that: include: Step 1: Acquire a source image and a target image, wherein the source image and the target image are multi-resolution digital pathology images obtained by scanning the same pathology section once by two different scanners; Step 2: At a low-resolution level, register the source image and the target image to obtain a first transformation matrix; Step 3: at a high-resolution level, based on the first transformation matrix, determining a second region in the source image corresponding to each first region of the target image, registering the first region and the second region, and obtaining a second transformation matrix; Step 4: Obtaining a registration result based on the first transformation matrix and the second transformation matrix; The step 4: obtaining a registration result based on the first transformation matrix and the second transformation matrix, comprises: Outputting and displaying the source image; Acquire a sight line change of a first user within a recently preset first time period when viewing the source image; Based on the change of the sight line, determining a moving trajectory of the sight line falling on the source image; Performing feature extraction on the movement trajectory based on a preset first feature extraction template to obtain a plurality of first feature values; Based on the first eigenvalue, construct a trajectory description vector; Get the preset trigger vector library; Matching the trajectory description vector with the trigger vector in the trigger vector library; If the match is met, determining at least one viewing area viewed by the user on the source image based on the movement trajectory; determining a registration region on the target image corresponding to the viewing region based on the first transformation matrix and the second transformation matrix; Based on the movement trajectory, determining a mapping area on the source image suitable for the registration area; The registration area is mapped into the mapping area to obtain a registration result, thereby completing the acquisition.

2. A multi-resolution digital pathology image registration method as claimed in claim 1, characterized in that: In step 2, registering the source image and the target image includes: Performing grayscale conversion on the source image to obtain a first grayscale image; Performing feature extraction on the first grayscale image to obtain first feature vectors of a plurality of first key points; Performing grayscale conversion on the target image to obtain a second grayscale image; Performing feature extraction on the second grayscale image to obtain second feature vectors of multiple second key points; Matching the first feature vector and the second feature vector; A first transformation matrix is ​​determined based on first coordinates of the first key point and second coordinates of the second key point corresponding to the first eigenvector and the second eigenvector that match each other.

3. A multi-resolution digital pathology image registration method as claimed in claim 1, characterized in that: In the step 3, registering the first region and the second region includes: Performing grayscale conversion on the first region to obtain a third grayscale image; Performing feature extraction on the third grayscale image to obtain third feature vectors of a plurality of third key points; Performing grayscale conversion on the second region to obtain a fourth grayscale image; Performing feature extraction on the fourth grayscale image to obtain fourth feature vectors of a plurality of fourth key points; Matching the third eigenvector and the fourth eigenvector; A second transformation matrix is ​​determined based on the third coordinates of the third key point and the fourth coordinates of the fourth key point corresponding to the third eigenvector and the fourth eigenvector that match each other.

4. A multi-resolution digital pathology image registration method as claimed in claim 1, characterized in that: The determining, based on the movement trajectory, at least one viewing area viewed by the user on the source image comprises: Acquire a plurality of first area blocks that the moving trajectory passes through in the source image; Traversing the first area blocks in sequence; During each traversal, extract the local trajectory of the moving trajectory falling within the traversed first area block, and at the same time, select a second area block adjacent to the traversed first area block in the source image except for other first area blocks; Based on a preset second feature extraction template, feature extraction is performed on the traversed first area block and the second area block and the relative position between the local trajectory and the first boundary line adjacent to the traversed first area block and the second area block to obtain a plurality of second feature values; Based on the second eigenvalue, construct an association description vector; Get the preset correlation vector library; Matching the association description vector with the association vector in the association vector library; If the match is met, the corresponding second area block is used as the third area block, and at the same time, the third area block is marked as unselectable; The first area block and the third area block are integrated to obtain a viewing area viewed by the user on the source image, thereby completing the determination.

5. A multi-resolution digital pathology image registration method as claimed in claim 1, characterized in that: Determining a mapping area on the source image suitable for the registration area based on the movement trajectory includes: Randomly mapping the registration area in the source image, and recording a plurality of first areas randomly mapped; taking the first area which has no overlap with the viewing area as the second area; Based on a preset third feature extraction template, feature extraction is performed on the relative position between the second area and the first area to obtain a plurality of third feature values; Based on the third eigenvalue, construct an analysis description vector; Acquire at least one preset analysis habit vector corresponding to the first user; Calculating the similarity between the analysis description vector and the analysis habit vector; If the similarity is greater than or equal to a preset similarity threshold, the corresponding second area is used as the third area; Based on a preset fourth feature extraction template, feature extraction is performed on a preset range around a plurality of points on the second boundary line of the third area in the source image to obtain a plurality of fourth feature values; Based on the fourth eigenvalue, construct a difference description vector; Obtaining a preset library of suitable evaluation vectors; Determining suitability based on the difference description vector and the suitability evaluation vector library; The third area corresponding to the maximum suitability is selected as a mapping area on the source image that is suitable for the registration area, and the determination is completed.

6. A multi-resolution digital pathology image registration method as claimed in claim 1, characterized in that: Also includes: Acquire analysis reference content corresponding to the viewing area and the registration area, and add the analysis reference content to the registration result; Wherein, obtaining analysis reference content corresponding to the viewing area and the registration area includes: Acquiring a preset analysis model corresponding to the slice type of the pathological slice; The viewing area and the registration area are input into the analysis model, and the analysis reference content is determined to complete the acquisition.

7. A multi-resolution digital pathology image registration method as claimed in claim 1, characterized in that: Also includes: When the first user requests at least one second user to jointly analyze the registration result, an online analysis room is constructed, and the first user and the second user are connected to the online analysis room; Publicly publishing the registration results in the online analysis room; Acquire multiple conversation records generated when the first user and the second user perform online analysis; Based on the conversation record, dynamically adjusting the registration result; The step of dynamically adjusting the registration result based on the conversation record includes: Construct a timeline; Based on the time point when the conversation record is generated, the conversation record is correspondingly set on the time axis; Performing semantic extraction on the conversation record to obtain a first semantic; Obtaining a preset adjustment instruction semantic library; Matching the first semantics with the second semantics in the adjustment instruction semantic library; If the match is met, based on the semantic understanding technology, based on the first semantics of a preset number of the conversation records after the conversation record corresponding to the first semantics that matches the match on the timeline, determining whether a preset adjustment operation corresponding to the second semantics that matches the match is effective; If so, in response to the adjustment operation, the registration result is adjusted accordingly.

8. A multi-resolution digital pathology image registration system, characterized in that: include: A first acquisition module, used for acquiring a source image and a target image, wherein the source image and the target image are multi-resolution digital pathology images obtained by scanning the same pathology slice once by two different scanners; A first registration module, used for registering the source image and the target image at a low resolution level to obtain a first transformation matrix; A second registration module is used to determine, at a high-resolution level, a second region in the source image corresponding to each first region of the target image based on the first transformation matrix, and to register the first region with the second region to obtain a second transformation matrix; A second acquisition module, used for acquiring a registration result based on the first transformation matrix and the second transformation matrix; The second acquisition module acquires a registration result based on the first transformation matrix and the second transformation matrix, including: Outputting and displaying the source image; Acquire a sight line change of a first user within a recently preset first time period when viewing the source image; Based on the change of the sight line, determining a moving trajectory of the sight line falling on the source image; Performing feature extraction on the movement trajectory based on a preset first feature extraction template to obtain a plurality of first feature values; Based on the first eigenvalue, construct a trajectory description vector; Get the preset trigger vector library; Matching the trajectory description vector with the trigger vector in the trigger vector library; If the match is met, determining at least one viewing area viewed by the user on the source image based on the movement trajectory; determining a registration region on the target image corresponding to the viewing region based on the first transformation matrix and the second transformation matrix; Based on the movement trajectory, determining a mapping area on the source image suitable for the registration area; The registration area is mapped into the mapping area to obtain a registration result, thereby completing the acquisition.

9. A multi-resolution digital pathology image registration system as claimed in claim 8, characterized in that: The first registration module registers the source image and the target image, including: Performing grayscale conversion on the source image to obtain a first grayscale image; Performing feature extraction on the first grayscale image to obtain first feature vectors of a plurality of first key points; Performing grayscale conversion on the target image to obtain a second grayscale image; Performing feature extraction on the second grayscale image to obtain second feature vectors of multiple second key points; Matching the first feature vector and the second feature vector; A first transformation matrix is ​​determined based on first coordinates of the first key point and second coordinates of the second key point corresponding to the first eigenvector and the second eigenvector that match each other.

Images